Format conversion of digital video signals, integration of digital video signals into photographic film material and the like, associated signal processing, and motion compensated interpolation of images

ABSTRACT

A system for integrating film material with a digital video signal employs a film scanner to produce a digital video signal from the source film, and a post production system for combining with that signal the input digital video signal. Motion interpolated temporal compensation is employed at stages of frame rate conversion.

BACKGROUND OF THE INVENTIONS Field of the Inventions

The first to fourth inventions disclosed herein relate to methods offormat conversion of digital video signals. The fifth to ninthinventions disclosed herein relate to methods of integrating digitalvideo signals into photographic film material. The tenth inventiondisclosed herein relates to associated signal processing apparatus. Theeleventh and twelfth inventions disclosed herein relate to methods ofmotion compensated temporal interpolation.

Description of the Prior Art Relevant to the First Invention

In video signaI production systems in which video signals are combinedand/or manipulated, it is necessary that any signals to be combined areof, or are converted to, the same format. Accordingly, format conversionof input. signals to the format of the production system and/or formatconversion of signals from the production system to a required outputformat becomes necessary for many applications.

In the case of integration of input 24 frame/s progressive scan formatmaterial with input 60 field/s 2:1 interlace scan format. material toproduce output material primarily in 24 frame/s progressive scan format,it would seem appropriate: (a) to convert the input 24 frame/sprogressive scan format to 60 field/s 2:1 interlace scan format; (b) forthe production system to operate on the two 60 field/s 2:1 interlacescan format signals to produce a combined 60 field/s 2:1 interlace scanformat signal; and (c) to convert the combined signal to 24 frame/sprogressive scan format. Preferably, the latter conversion `c` shouldemploy motion compensated temporal interpolation, as described in UnitedKingdom patent application GB 2231228A in view of the change of framerate. Also, the former conversion `a` from 24 frame/s progressive scanformat to 60 field/s 2:1 interlace scan format should employ motioncompensated interpolation, if such is possible, again in view of thechange of frame rate. However, there is a risk that motion compensatedinterpolation may introduce artifacts into the output image, and withtwo stages of motion compensated interpolation, as suggested above, suchartifacts may prove to be unacceptable.

OBJECT AND SUMMARY OF THE FIRST INVENTION

An aim of the first invention is to provide a conversion method whichmay be used in an overall system as described above, but which may alsohave other applications, which obviates the need for two stages ofmotion compensated interpolation in such a system.

In accordance with the first invention, there is provided a method ofconverting an input 60 field/s 2:1 interlace scan format digital videosignal to an output 24 frame/s 60 field/s 3232 pulldown format digitalvideo signal, comprising the steps of:

forming a first series of 60 frame/s progressive scan format frames fromthe fields of the input signal;

forming a second series of 24 frame/s progressive scan format framesfrom the first series of frames such that at least every other frame inthe second series of frames is produced by motion compensatedinterpolation between a respective pair of successive frames of thefirst series of frames; and

outputting alternately odd and even fields from the second series offrames such that one field in every five is a repeat.

Such a 3232 pulldown format, as described in United Kigdom patentapplication GB 9018805.3 involves producing, from each of a series offour frames of the 24 frame/s progressive scan format frames, an odd andan even field, and also producing from the first arid third framesadditional odd and even fields, respectively, referred to as `phantomfields`, which may then be ignored at the end of the production process.Accordingly, four frames in the 24 frame/s progressive scan formatbecome ten fields in the 24 frame/s 60 field/s 3232 pulldown format.

By employing the conversion method of the first invention in an overallsystem as described above: (a) the input 24 frame/s progressive scanformat material can be converted to 24 frame/s 60 field/s 3232 pulldownformat without requiring motion compensated interpolation because thereis no change of frame rate; (b) the input 60 field/s 2:1 interlace scanformat signal can be converted to 24 frame/s 60 field/s 3232 pulldownformat using the method according to the first invention and employingmotion compensated interpolation as required; (c) the production systemcan operate on the two 24 frame/s 60 field/s 3232 pulldown formatsignals; and (d) the output signal from the production system in 24frame/s 60 field/s 3232 pulldown format can be converted to 24 frame/sprogressive scan format employing a drop field technique which drops thephantom fields but which does not require motion compensatedinterpolation because there is no change in frame rate. Accordinglybetween the two sources and the primary output there is only one stageof motion compensated interpolation.

In one example of the first invention alternate frames of the secondseries are produced by motion compensated interpolation equally betweena respective successive pair of frames of the first series, and theremaining frames of the second series are not produced by motioninterpolated compensation. Accordingly, alternate frames of the secondseries are equally temporally offset between pairs of frames of thefirst series, and the other frames of the second series are temporallyaligned with frames of the first series.

In one form, the output fields are output at a rate of 5/2 times therate of production of the frames in the second series of frames. Thisenables the required order of the output fields to equate with the orderof their production.

In an alternative form, the. frames in the second series are stored atthe same rate as they are produced, and the output fields are producedfrom the stored frames such that one field in every five is a repeat.This requires some re-ordering of the output fields, but may requireless storage space to be used than in other examples.

In a preferred example of the first invention each input field isrepeated with a repeat rate 4R (where R is an integer, for example 2)times, a frame is produced in the first series for every 4R repeatedinput fields, a frame is produced in the second series for every 10Rrepeated input fields, and each frame in the second series is repeatedwith a repeat rate of 5R times. This causes the conversion process to beslowed down to less than real-time rate, as may be required by thecomputational intensity of the conversion and interpolation methods.

An embodiment of the first invention is described particularly withreference to FIGS. 49 to 51 and 66 of the accompanying drawings.

Description of the Prior Art Relevant to the Second Invention

In the case of integration of input 30 frame/s progressive scan formatmaterial with input 60 field/s 2:1 interlace scan format material toproduce output: material primarily in 24 or 30 frame/s progressive scanformat, it would seem appropriate: (a) to convert the input 30 frame/sprogressive scan format to 60 field/s 2:1 interlace scan format; (b) forthe product ion system to operate on the two 60 field/s 2:1 interlacescan format signals to produce a combined 60 field/s 2:1 interlace scanformat signal, because currently available post-production equipmentrequires the video signals to be interlaced; and (c) to convert thecombined signal to 24 or 30 frame/s progressive scan format. Preferably,the latter conversion `c`, in the case of 24 frame/s output, shouldemploy motion compensated interpolation, as described in the aforesaidpatent application GB 2231228A in view of the change of frame rate.Also, the former conversion `a` from 30 frame/s progressive scan formatto 60 field/s 2:1 interlace scan format should employ motion compensatedinterpolation, if such is possible, again in view of the change of framerate. However, there is a risk that motion compensated interpolation mayintroduce artifacts into the output: image, and with two stages ofmotion compensated interpolation, as suggested above, such artifacts mayprove to be unacceptable.

OBJECT AND SUMMARY OF TItE SECOND INVENTION

An aim of the second invention is to provide a conversion method whichmay be used in an overall system as described above, but which may alsohave other applications, which obviates the need for two stages ofmotion compensated interpolation in such a system.

In accordance with the second invention, there is provided a method ofconverting an input 60 field/s 2:1 interlace format digital video signalinto an out. put 30 frame/s progressive scan format digital videosignal, comprising the steps of:

forming a series of progressive scan format frames from the inputfields; and

forming the output. frames by motion compensated interpolation betweenrespective pairs of successive frames of the series of progressive scanformat frames.

By employing the conversion method of the second invention in an overallsystem as described above: (a) the input 30 frame/s progressive scanformat material needs no conversion; (b) the input 60 field/s 2:1interlace scan format signal can be converted to 30 frame/s progressivescan format using the method according to the invention and employingmotion compensated interpolation as required; (c) the production systemcan operate on the two 30 frame/s progressive scan format signals; and(d) the output, signal from the production system in 30 frame/sprogressive scan format can be directly used or can be converted to 24frame/s progressive scan format employing motion compensatedinterpolation.

A production system which can operate on progressive scan format signalsis described in the aforesaid patent application GB 9018805.3, thecontent of which is incorporated herein by reference.

Although, using the conversion method of the second invention in asystem as described above employs motion compensated in step `b` andalso in step `d` for 24 frame/s progressive scan format output, themethod of the invention preferably also includes the step of selectingwhether to form the output frames by motion compensated interpolationbetween respective pairs of successive frames of said series, or whetherto form the output frames directly from alternate frames in said series.This is because when the source picture is noisy or there is anincorrect assessment in the formation of the progressive scan formatframes, the picture will lose vertical detail and alias components willbe present. However, when these problems do not arise, deselection ofmotion compensated interpolation will provide a satisfactory outputimage and in this case there will be no stages of motion compensatedinterpolation for 30 frame/s progressive scan format output, and onlyone stage for 24 frame/s progressive scan format output.

In the method of the second invention, preferably each output frame isinterpolated half-way between the respective frames in said series ofprogressive scan format frames, and preferably a respective progressivescan format frame is formed in said series for each input field.

Also, the progressive format frames in said series are preferably formedusing motion adaptive interpolation to produce each frame by intrafieldinterpolation within a respective input field and/or by interframeinterpolation between successive input frames.

An embodiment of the second invention is described particularly withreference to FIGS. 57 to 60 and 67 to 70 of the accompanying drawings.

Description of the Prior Art Relevant to the Third Invention

In the case of integration of input 30 frame/s progressive scan formatmaterial with input 60 field/s 2:1 interlace scan format material toproduce output material primarily in 60 field/s 2:1 interlace scanformat, it would seem appropriate: (a) to convert the input 30 frame/sprogressive scan format to 60 field/s 2:1 interlace scan format; and (b)for the production system to operate on the two 60 field/s 2:1 interlacescan format signals to produce a combined 60 field/s 2:1 interlace scanformat signal.

The conversion from 30 frame/s progressive scan format to 60 field/s 2:1interlace scan format involves a change in field rate, and if theconversion is carried out by forming pairs of output fields from thesame input frame, then defects will arise in the output picture, such asjudder.

OBJECT AND SUMMARY OF THE THIRD INVENTION

In order to deal with this problem, the third invention provides amethod of converting an input 60 field/s 2:1 interlace scan formatdigital video signal in which the fields of each field pair in the inputsignal represent respective temporally identical portions of the inputimage into an output 60 field/s 2:1 interlace scan format digital videosignal in which the fields of each field pair in the output signalrepresent respective temporally offset portions of the output image,comprising the steps of:

forming a series of progressive scan format frames from the field pairsof the input signal; and

forming the output fields from the progressive scan format frames suchthat at least every other output field is produced by motion compensatedinterpolation between a respective pair of successive frames of theseries of progressive scan format frames.

As mentioned before, mot ion compensated interpolation is described inpatent application GB2231228A, but the method described therein relatesonly to interpolation in the case where the field rate is decreased,that is from 60 field/s 2:1 interlace scan format to 24 frame/sprogressive scan format. In the method of the third invention, the fieldrate is increased, and this requires more complex techniques of motioncompensated interpolation.

Preferably, every output field is produced by motion compensatedinterpolation between a respective pail of successive frames of theseries of progressive scan format frames. By virtue of this feature,noise modulation in the output image is reduced. Specifically, themethod preferably provides that:

one field of every output field pair is produced by motion interpolatedcompensation one-quarter of the way between a respective pair ofsuccessive frames of the series of progressive scan format frames; and

the other field of every output field pair is produced by motioninterpolated compensation three-quarters of the way between that pair ofsuccessive frames of the series of progressive scan format frames.

By virtue of this feature, the interpolated spatial response of thefield pairs is evened out.

The method may further comprise the steps of:

repeating each input field at a repeat rate of N times (where N isgreater than 1);

producing a respective frame of the series of progressive scan formatframes for every 2N repeated input fields;

using each progressive scan format frame in contributing to fourinterlace scan format fields;

repeating each interlace scan format field at a repeat rate of N times;and

outputting one in every N interlace scan format fields as a respectiveoutput field.

This feature causes the conversion process to be slowed clown to lessthan real-time rate, as may be required by the computational intensityof the conversion and interpolation processes.

The method may further comprise the step of generating the input 60field/s 2:1 interlace scan format digital video signal from sourcematerial in 30 frame/s format such that the fields of each field pair ofthe input signal are produced from the same respective frame of thesource material.

An embodiment of the third invention is described particularly withreference to FIGS. 52 to 54 and 68 to 70 of the accompanying drawings.

Description of the Piror Art Relevant to the Fourth Invention

In the case of integration of input 24 frame/s progressive scan formatmaterial with input 60 field/s 2:1 interlace scan format material toproduce output material primarily in 60 field/s 2:1 interlace scanformat, it would seem appropriate: (a) to convert the input 24 frame/sprogressive scan format to 60 field/s 2:1 interlace scan format; and (b)for the production system to operate on the two 60 field/s 2:1 interlacescan format signals to produce a combined 60 field/s 2:1 interlace scanformat signal . Also, in the case of integration of 24 frame/sprogressive scan format material with 30 frame/s progressive scan formatmaterial to produce 30 frame/s progressive scan format materiaI, itwould seem appropriate: (a) to convert the input 24 frame/s progress ivescan format to 30 frame/s progressive scan format material; and (b) forthe production system to operate on the two 30 frame/s progressive scanformat signals to produce a combined 30 frame/s progressive scan formatsignal.

The conversions from 24 frame/s progressive scan format to 60 field/s2:1 interlace scan format or 30 frame/s progressive scan format involvea change in frame rate, and if the conversion is carried out by formingeach output field or frame from the input frame which is temporallyclosest to that output field or frame, then defects will arise in theoutput picture, such as judder.

OBJECT AND SUMMARY OF THE FOURTH INVENTION

In order to deal with this problem, one aspect of the present inventionprovides a method of converting an input 24 frame/s progressive scanformat digital video signal into an output 30 frame/s progressive scanformat digital video signal, comprising the steps of forming the outputframes from the input frames such that at least four of every fiveoutput frames are produced by motion compensated interpolation betweensuccessive pairs of the input frames.

Another aspect of the present invention provides a method of convertingan input 24 frame/s progressive scan format digital video signal into anoutput 30 frame/s progressive scan format digital video signal,comprising the steps of forming the output frames from the input framessuch that at least four of every five output frames are produced bymotion compensated interpolation between successive pairs of the inputframes.

As mentioned above, motion compensated interpolation is described inpatent application GB2231228A, but the method described therein relatesonly to interpolation in the case where the frame rate is decreased,that is from 60 field/s 2:1 interlace scan formal to 24 frame/s progressive scan format. In the methods of the first and second aspects of thefourth invention, the frame rate is increased, and this requires morecomplex techniques of motion compensated interpolation.

Specifically, in the method of the first aspect of the fourth invention,preferably a series of five successive output fields are produced from aseries of three successive input frames such that:

a) the first output field is produced from the first input frame notnecessarily with motion compensated interpolation;

b) the second output: field is produced by motion compensatedinterpolation two-fifths of the way between the first and second inputframes;

c) the third output field is produced by motion compensatedinterpolation four-fifths of the way between the first and second inputframes;

d) the fourth output field is produced by motion compensatedinterpolation one-fifth of the way between the second and third inputframes; and

e) the fifth output field is produced by motion compensatedinterpolation three-fifths of the way between the second and third inputframes.

Alternatively, in the method of the second aspect of the fourthinvention, preferably a series of five successive output frames areproduced from a series of five successive input frames such that:

a) the first output frame is produced from the first input frame notnecessarily with motion compensated interpolation;

b) the second output frame is produced by motion compensatedinterpolation four-fifths of the way between the first and second inputframes;

c) the third output frame is produced by motion compensatedinterpolation three-fifths of the way between the second and third inputframes;

d) the fourth output frame i s produced by mot ion compensatedinterpolation two-fifths of the way between the third and fourth inputframes; and

e) the fifth output frame is produced by motion compensatedinterpolation one-fifth of the way between the fourth and fifth inputframes.

Either method may further comprise the steps of repeating in a firstseries each input frame 5R times (where R is an integer such as 5),producing a new frame in a second series in the 60 field/s 2:1 interlaceformat for every 4R repeated input frames, repeating each new frame inthe second series with a repeat rate of 4R times, and outputting one inevery 4R fields of the second series of frames as a respective outputfield. This causes the conversion process to be slowed down to less thanreal-time rate, as may be required by the computational intensity of theconversion and interpolation methods.

In the case where the input frames are provided as 2:1 interlace fieldpairs in which the fields of each pair represent respective temporallyidentical portions of time input image, time fields of each input fieldpair are preferably each repeated with a repeat rate 5R times, anintermediate series of progressive scan format frames being producedfrom the input fields with each progressive scan format frame beingproduced from a respective pair of the fields, and the output fieldsbeing formed from the progressive scan format frames.

In the case of the method of the first aspect of the fourth invention,if the input frames are provided in 60 field/s 2:1 interlaced 3232pulldown format (as described in the aforementioned patent applicationGB 9018805.3), the method may further comprise the steps of repeating ina first series each input field 4R times, producing once for every 10Rrepeated input fields a new progressive scan format frame in anintermediate series from pairs of successive non-repeat input fields,producing from the intermediate series a new frame in a second series inthe 60 field/s 2:1 interlace format for every 4R repeated input frames,repeating each new frame in the second series with a repeat rate of 4Rtimes, and outputting one in every 4R fields of the second series offrames as a respective output field.

An embodiment of the fourth invention is described particularly withreference to FIGS. 55, 56 and 67 of the accompanying drawings.

OBJECT AND SLiMMARY OF THE FIFTH INVENTION

In accordance with the fifth invention, there is provided a method ofintegrating input 24 frame/s format material with an input 60 field/s2:1 interlace scan format digital video signal to produce an output 60field/s 2:1 interlace scan format digital video signal, comprising thesteps of:

(a) producing from the input 24 frame/s format material a 24 frame/sprogressive scan format digital video signal;

(b) converting the 24 frame/s progressive scan format digital signal toa 60 field/s 2:1 interlace scan format digital video signal; and

(c) combining the signal so produced with the input 60 field/s interlacescan format digital video signal to produce a combined digital videosignal.

Preferably, in conversion step `b`, at least some of the fields of theconverted signal are produced using motion compensated interpolationbetween successive frames of the 24 frame/s progressive scan formatdigital video signal.

As mentioned above, moLlon compensated interpolation is described inpatent application GB2231228A, but the method described therein relatesonly to interpolation in the case where the frame rate is decreased,that is from 60 field/s 2:1 interlace scan format to 24 frame/sprogressive scan format. In the methods of the fifth invention, theframe rate is increased, and this requires more complex techniques ofmotion compensated interpolation.

Preferably, production step `a` comprises the step of scanning 24frame/s source material to produce the 24 frame/s progressive scanformat digital video signal.

The method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to 30 frame/sprogressive scan format. In this case, in the conversion to 30 frame/sprogressive scan format, preferably none of the frames of the convertedsignal are produced using motion compensated interpolation betweensuccessive fields of the combined 60 field/s 2:1 interlace scan formatdigital video signal. Thus, two stages of motion compensatedinterpolation between a primary source and output are avoided. Thisexample of the method may further comprise the steps of:

digital-to-analogue converting the 30 frame/s progressive scan formatdigital video signal to form a 30 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 30frame/s format.

The method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to 24 frame/sprogressive scan format. In this case, in the conversion to 24 frame/sprogressive scan format, at least some of the frames of the convertedsignal may be produced using motion compensated interpoiation betweensuccessive fields of the combined 60 field/s 2:1 interlace scan formatdigital video signal. This example of the method may further comprisethe steps of:

digital-to-analogue converting the 24 frame/s progressive scan formatdigital video signal to form a 24 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 24frame/s format.

The method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to at least oneof the following formats:

NTSC format;

50 frame/s 1250-lines progressive scan format;

50 field/s 1250-lines 2:1 interlace scan format; and

50 field/s 625-lines 2:1 interlace scan format.

In this case, a further step may be provided of converting the1250-lines format signal to 50 field/s 625-lines 2:1 interlace scanformat.

An embodiment of the fifth invention is described particularly withreference to FIGS. 55, 56 and 67 of the accompanying drawings.

Description of the Prior Art Relevant to the Sixth Invention

In the case of integration of input 24 frame/s progressive scan formatmaterial (which may be derived from 24 frame/s format photographic film)with input 60 field/s 2:1 interlace scan format material to produceoutput material primarily in 24 frame/s progressive scan format, itwould seem appropriate: (a) to convert the input 24 frame/s progressivescan format to 60 field/s 2:1 interlace scan format; (b) for theproduction system to operate on the two 60 field/s 2:1 interlace scanformat signals to produce a combined 60 field/s 2:1 interlace scanformat signal; and (c) to convert the combined signal to 24 frame/sprogressive scan format. Preferably, the latter conversion `c` shouldemploy motion compensated interpolation, as described in the aforesaidpatent application GB2231228A in view of the change of frame rate. Also,the former conversion `a` from 24 frame/s progressive scan format to 60field/s 2:1 interlace scan format should employ motion compensatedinterpolation, if such is possible, again in view of the change of framerate. However, there is a risk that motion compensated interpolation mayintroduce artifacts into the output image, and with two stages of motioncompensated interpolation, as suggested above, such artifacts may proveto be unacceptable.

OBJECT AND SUMMARY OF THE SIXTH INVENTION

An aim of the sixth invention is to provide a method which obviates theneed for two stages of motion compensated interpolation.

In accordance with the sixth invention, there is provided a method ofintegrating input 24 frame/s progressive scan format material with aninput 60 field/s 2:1 interlace scan format digital video signal toproduce output 24 frame/s progressive scan format material, comprisingthe steps of:

(a) producing from the input progressive scan material a 60 field/spulldown format digital video signal with phantom fields;

(b) converting the input 2:1 interlace scan format video signal to the60 field/s pulldown format with phantom fields;

(c) combining the two video signals in 60 field/s pulldown format toproduce a combined digital video signal; and

(d) producing the output progressive scan material from the nonphantomfields of the combined digital video signal.

The pulldown format may be a 3232 pulldown format, as described in theaforementioned patent application GB 9018805.3. This involves producing,from each of a series of four frames of the 24 frame/s progressive scanformat frames, an odd and an even field, and also producing from thefirst and third frames additional odd and even fields, respectively,referred to as `phantom fields`, which may then be ignored at the end ofthe production process. Accordingly, four frames in the 24 frame/sprogressive scan format become ten fields in the 24 frame/s 60 field/s3232 pulldown format.

By employing the met hod of the sixth invention as described above: (a)the input 24 frame/s progressive scan format material can be convertedto 24 frame/s 60 field/s pulldown format without requiring motioncompensated interpolation because there is no change of frame rate; (b)the input 60 field/s 2:1 interlace scan format signal can be convertedto 24 frame/s 60 field/s pulldown format using the method according tothe sixth invention and employing motion compensated interpolation asrequired; (c) the production system can operate on the two 24 frame/s 60field/s pulldown format signals; and (d) the output signal from theproduction system in 24 frame/s 60 field/s pulldown format can beconverted to 24 frame/s progressive scan format employing a drop fieldtechnique which drops the phantom fields but which does not requiremotion compensated interpolation because there is no change in framerate. Accordingly between the two sources and the primary output thereis only one stage of motion compensated interpolation.

Production step `a` may comprise the steps of:

scanning 24 frame/s format source material film to produce a 24 frame/sprogressive scan format digital video signal; and

producing from the 24 frame/s progressive scan format digital signal the60 field/s pulldown format video signal having pairs of fieldsrepresenting portions of the same frame and the phantom fields.

Production step `d` may comprise the steps of:

dropping the phantom fields from the combined digital signal andcombining pairs of fields of the resultant signal to produce a 24frame/s progressive scan format digital video signal;

digital-to-analogue converting the 24 frame/s progressive scan formatdigital video signal to form a 24 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film.

In this case, the method may further comprise the step of converting the24 frame/s progressive scan format digital video signal to at least oneof the following formats:

pseudo-50 field/s 625-lines 2:1 interlace scan format;

pseudo-50 frame/s progressive scan high-definition format; and

pseudo-50 field/s 2:1 interlace scan format.

If desired for any purpose, the method may further comprise the step ofoutputting the combined pulldown format digital video signal.

The method may further comprise the step of converting the combinedpulldown format digital video signal to at least one of the followingformats:

NTSC format; and

60 field/s 2:1 interlace scan high-definition digital video format.

In this case, the method may further comprise the step of converting the60 field/s 2:1 interlace scan high-definition format digital videosignal to NTSC format.

An embodiment of the sixth invention is described particularly withreference to FIGS. 49 to 51 and 66 of the accompanying drawings.

OBJECT AND SUMMARY OF THE SEVENTH INVENTION

In accordance with the seventh invention there is provided a method ofintegrating input 30 frame/s format material and a 60 field/s 2:1interlace scan format digital video signal to produce an output 60field/s 2:1 interlace scan format digital video signal, comprising thesteps of:

(a) producing from the input 30 frame/s format material a 30 frame/sprogressive scan format digital video signal;

(b) converting the 30 frame/s progressive scan format digital signal toa 60 field/s 2:1 interlace scan format digital video signal; and

(c) combining the converted signal with the input 60 field/s interlacescan format digital video signal to produce a combined digital videosignal.

The conversion from 30 frame/s progressive scan format to 60 field/s 2:1interlace scan format involves a change in field rate, and if theconversion is carried out by forming pairs of output fields from thesame input frame, then defects will arise in the output picture, such asjudder.

In order to deal with this problem, in conversion step `b`, at leastsome of the fields of the converted signal are produced using motioncompensated interpolation between successive frames of the 30 frame/sprogressive scan format digital video signal.

In one example of the method, production step `a` comprises the step ofscanning 30 frame/s format source material to produce the 30 frame/sprogressive scan format digital video signal.

The method according to the seventh invention may also be used tointegrate input 60 frame/s format material and the 60 field/s 2:1interlace scan format dig tal video signal to produce the output 60field/s 2:1 interlace scan format digital video signal, and in this casefurther comprises the steps of:

producing from the input 60 frame/s format material a 60 frame/sprogressive scan format digital video signal;

converting the 60 frame/s progressive scan format digital signal to a 60field/s 2:1 interlace scan format digital video signal by a pull-downevery field technique; and

combining the converted signal with the input 60 field/s interlace scanformat dig tal video signal to produce the combined digital videosignal.

The method may optionally comprise the step of converting the combined60 field/s 2:1 interlace scan format digital video signal to 30 frame/sprogressive scan format. In this case, in the conversion to 30 frame/sprogressive scan format, preferably none of the frames of the convertedsignal are produced using motion compensated interpolation betweensuccessive fields of the combined 60 field/s 2:1 interlace scan formatdigital video signal. This example of the method may further comprisethe steps of:

digital-to-analogue converting the 30 frame/s progressive scan formatdigital video signal to form a 30 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 30frame/s format.

The method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to 24 frame/sprogressive scan format. In this case, in the conversion to 24 frame/sprogressive scan format, at least some of the frames of the convertedsignal are preferably produced using motion compensated interpolationbetween successive fields of the combined 60 field/s 2:1 interlace scanformat digital video signal. This example of the method may furthercomprise the steps of:

digital-to-analogue converting the 24 frame/s progressive scan formatdigital video signal to form a 24 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 24frame/s format.

The method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to at least oneof the following formats:

NTSC format;

50 frame/s 1250-1ines progressive scan format;

50 field/s 1250-lines interlace scan format; and

50 field/s 625-lines 2:1 interlace scan format.

In this case the method may further comprise the step of converting the1250-lines format signal to 50 field/s 625-1ines 2:1 interlace scanformat.

An embodiment of the seventh invention is described particularly withreference to FIGS. 52 to 54 and 69 of the accompanying drawings.

Description of the Prior Art Relevant to the Eighth Invention

In the case of integration of input 30 frame/s progressive scan formatmaterial with input 30 frame/s progressive scan format: video signalsand optionally with input 60 field/s 2:1 interlace scan format materialto produce output material primarily in 30 frame/s progressive scanformat, it would seem appropriate: (a) to convert the input 30 frame/sprogressive scan format signals to 60 field/s 2:1 interlace scan format;(b) for the production system to operate on the 60 field/s 2:1 interlacescan format signals to produce a combined 60 field/s 2:1 interlace scanformat signal, because currently available postproduction equipmentrequires the video signals to be interlaced; and (c) to convert thecombined signal to 30 frame/s progressive scan format. The formerconversion `a` from 30 frame/s progressive scan format to 60 field/s 2:1interlace scan format should employ motion compensated interpolation, ifsuch is possible, in view of the change of field rate. However, there isa risk that motion compensated interpolation may introduce artifactsinto the output image.

OBJECT AND SUMMARY OF THE EIGHTH INVENTION

An aim of the eighth invention is to provide a method generally asdescribed above, which obviates the need for motion compensatedinterpolation between the primary sources and output.

In accordance with the eighth invention, there is provided a method ofintegrating input 30 frame/s progressive scan format material with aninput 30 frame/s progressive scan format digital video signal to produceoutput 30 frame/s progressive scan format material, comprising the stepsof:

(a) producing from the input progressive scan material a 30 frame/sprogressive scan format digital video signal;

(b) combining the 30 frame/s progressive scan format signal with theinput 30 frame/s progressive scan format signal to produce a combined 30frame/s progressive scan format digital video signal; and

(c) producing the output progressive scan material from the combineddigital video signal.

By employing this method: (a) the input 30 frame/s progressive scanformat material needs no conversion; (b) any optional input 60 field/s2:1 interlace scan format signal can be converted to 30 frame/sprogressive scan format using a methoot which does not necessarilyrequire motion compensated interpolation; (c) the production system canoperate on the two 30 frame/s progressive scan format signals; and (d)the output signal from the production system in 30 frame/s progressivescan format can be directly used.

A production system which can operate on progressive scan format signalsis described in the aforementioned patent application GB 9018805.3.

In view of the above, the method optionally includes the step ofconverting an input 60 field/s 2:1 interiace scan format digital videosignal to 30 frame/s progressive scan format to form the or an input 30frame/s progressive scan format digital video signal. In this case, inthe conversion from the 60 field/s 2:1 interlace scan format signal tothe 30 frame/s progressive scan format signal, preferably none of theframes of the converted signal are produced using motion compensatedinterpolation between successive fields of the input 60 field/s 2:1interlace scan format signal.

Production step `c` may comprise the step of scanning 30 frame/s formatsource material to produce the 30 frame/s progressive scan formatdigital video signal.

Production step `c` may comprise the steps of:

digital-to-analogue converting the 30 frame/s progressive scan formatdigital video signal to form a 30 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film.

The method may further comprise the step of converting the combined 30frame/s progressive scan format signal to a 60 field/s 2:1 interlacescan format signal. In this case, at least some of the fields of the 60field/s 2:1 interlace scan format signal may be produced using motioncompensated interpolation between successive frames of the combined 30frame/s progressive scan format signal. Furthermore, this example of themethod may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to 24 frame/sprogressive scan format. In this case, in the conversion to 24 frame/sprogressive scan format, at least some of the frames of the convertedsignal may be produced using motion compensated interpolation betweensuccessive fields of the combined 60 field/s 2:1 interlace scan formatdigital video signal. The method may further comprise the steps of:

digital-to-analogue converting the 24 frame/s progressive scan formatdigital video signal to form a 24 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 24frame/s format.

If a conversion to 60 field/s 2:1 interlace scan format is provided, themethod may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to at least oneof the foliowing formats:

NTSC format;

50 frame/s 1250-lines progressive scan format;

50 field/s 1250-1ines 2:1 interlace scan format; and

50 field/s 625-lines 2:1 interlace scan format.

In this case, the method may furtimer comprise the step of convertingthe 1250-lines format signal to 50 field/s 625-lines 2:1 interlace scanformat.

An embodiment of the eighth invention is described particularly withreference to FIGS. 57, 58 and 68 of the accompanying drawings.

Description of the Prior Art Relevant to the Ninth Invention

In the case of integration of input 30 frame/s progressive scan formatmaterial (such as photographic film) with input 30 frame/s progressivescan format material to produce output material primarily in 24 frame/sprogressive scan formaL, it would seem appropriate: (a) to convert theinput 30 frame/s progressive scan formats to 60 field/s 2:1 interlacescan format; (b) for the production system to operate on the two 60field/s 2:1 interlace scan format signals to produce a combined 60field/s 2:1 interlace scan formal signal, because currently availablepost-production equipment requires the video signals to be interlaced;and (c) to convert the combined signal to 24 frame/s progressive scanformat. Preferably, the latter conversion `c`, in the case of 24 frame/soutput, would employ motion compensated interpolation, as described inthe aforementioned patent application GB2231228A in view of the changeof frame rate. Also, the former conversion `a` from 30 frame/sprogressive scan format to 60 field/s 2:1 interlace scan format shouldemploy motion compensated interpolation, if such is possible, again inview of the change of frame rate. However, there is a risk that motioncompensated interpolation may introduce artifacts into the output image,and with two stages of motion compensated interpolation, as suggestedabove, such artifacts may prove to be unacceptable.

OBJECT AND SUMMARY OF THE NINTH INVENTION

An aim of the ninth invention is to provide a method generally asdescribed above, which obviates the need for two stages of motioncompensated interpolation.

In accordance with the ninth present invention, there is provided amethod of integrating input 30 frame/s progressive scan format materialwith an input 30 frame/s progressive scan format digital video signal toproduce output 24 frame/s progressive scan format material, comprisingthe steps of:

(a) producing from the input progressive scan material a 30 frame/sprogressive scan format digital video signal;

(b) combining the 30 frame/s progressive scan format signal with theinput 30 frame/s progressive scan format signal to produce a combined 30frame/s progressive scan format digital video signal;

(c) converting the combined 30 frame/s progressive scan format videosignal to a 24 frame/s progressive scan format digital video signalemploying motion compensated interpolation; and

(d) producing the output 24 frame/s progressive scan material from thecombined digital video signal.

By this method: (a) the input 30 frame/s progressive scan format signalsneed no format conversion; (b) the production system operates on the two30 frame/s progressive scan format signals; and (c) the output signalfrom the production system in 30 frame/s progressive scan format isconverted to 24 frame/s progressive scan format employing motioncompensated interpolation. Thus, there is only one stage of motioncompensated interpolation between the primary inputs and outputs.

As mentioned above, a production system which can operate on progressivescan format signals is described in patent application GB 9018805.3, thecontent of which is incorporated herein by reference.

The method may further comprise the step of converting an input 60field/s 2:1 interlace scan format digital video signal to 30 frame/sprogressive scan format to form the input 30 frame/s progressive scanformat digital video signal. In this case, in the conversion from the 60field/s 2:1 interlace scan format signal to the 30 frame/s progressivescan format signal, preferably none of the frames of the convertedsignal are produced using motion compensated interpolation betweensuccessive fields of the input 60 field/s 2:1 interlace scan formatsignal.

Production step `a` may comprise the step of scanning 30 frame/s formatsource material to produce the 30 frame/s progressive scan formatdigital video signal.

Production step `d` may comprise the steps of:

digital-to-analogue converting the 24 frame/s progressive scan formatdigital video signal to form a 24 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 24frame/s format.

The method may further comprise the steps of:

digital-to-analogue converting the 30 frame/s progressive scan formatdigital video signal to form a 30 frame/s progressive scan formatanalogue video signal; and

supplying the analogue video signal to a photographic recorder to recordthe content of the analogue video signal on photographic film in 30frame/s format.

The method may also t urther comprise the step of converting thecombined 30 frame/s progressive scan format signal to a 60 field/s 2:1interlace scan format signal. In this case, at least some of the fieldsof the 60 field/s 2:1 interlace scan format signal are preferablyproduced using motion compensated interpolation between successiveframes of the combined 30 frame/s progressive scan format signal. Also,the method may further comprise the step of converting the combined 60field/s 2:1 interlace scan format digital video signal to at least oneof the following formats:

NTSC format;

50 frame/s 1250-lines progressive scan format;

50 field/s 1250-lines 2:1 interlace scan format; and

50 field/s 625-lines 2:1 interlace scan format.

In this case, the method may further comprise the step of converting the1250-lines format signal to 50 field/s 625-lines interlace scan format.

An embodiment of the ninth invention is described particularly withreference to FIGS. 70 of the accompanying drawings.

Description of the Prior Art Relevant to the Tenth Invention

Aforementioned patent application GB2231228A describes an arrangement inwhich a VTR plays at one-eighth speed into a standards converter, andthe standards converter provides ten repeats of each output frame. Aframe recorder stores one in every ten of the repeated output framesuntil it is full, and the stored frames are then output at normal speedto a VTR which records at normal speed. The material is thereforeconverted in segments, entailing starting, stopping and cuing of both ofthe VTRs. In order to convert one hour of source material using a framerecorder with a capacity of 256 frames, it is necessary to start, stopand cue each of the recorders 338 times, and it will be realised thatthis can cause considerable wear of both recorders. Furthermore, theoperations of alternately reading from the input VTR and then recordingon the output VTR, with cuing of both recorders results in theconversion process being slow. Indeed, in the example given above,although the standards converter processes at one-eighth speed, theconversion of one hour of material would take not 8 hours, but almost 11hours. With smaller capacity frame recorders, the wasted the would beincreased.

OBJECT AND SUMMARY OF THE TENTH INVENTION

An aim of a first aspect of the tentim invention is to increase theconversion rate of the arrangement described above.

In accordance with the first aspect of the tenth invention, there isprovided a signal processing apparatus, comprising:

source means for outputting at a first signal rate a first signal havinga second inherent signal rate which is faster than the first signalrate;

processing means for processing the first signal at the first rate andoutputting at the first signal rate a second processed signal having athird inherent signal rate which is faster than the first signal rate;

post-processing storage means for temporarily storing the second signalat said first signal rate and for outputting the stored signal as athird signal at the faster third signal rate; and

recording means for recording the third signal at the third signal rate;

wherein the storage means is operable to store the second signalcontinuously and to output: the third signal intermittently and at thesame the as storing the second signal.

It can therefore be arranged that the source means and the processingmeans operate non-stop.

In one embodiment, the storage means comprises:

at least two sub-storage means; and

control means for controlling the sub-storage means such that portionsof the second signal are stored alternately in the sub-storage means,and such that, while a portion of the second signal is being stored inone of the sub-storage means, a portion of the third signal can beoutput from the other, or one of the other, sub-storage means.

However, in another embodimenL the storage means is operable:

to store the second signal continuously at storage locations which arecycled continuously in the storage means;

to output the third signal at the same time as the second signal isbeing stored and intermittently such that no portion of the storedsignal is overwritten before it is output as a portion of the thirdsignal.

This latter embodiment requires less storage capacity in the storagemeans.

The arrangement described in GB2231228A requires a VTR capable ofproviding a slow-motion output. An aim of a second aspect of the tenthinvention is to obviate the need for a slow-motion VTR, or like source.

In accordance with the second aspect of the tenth invention, there isprovided a signal processing apparatus, comprising:

source means for intermittently outputting a fourtim signal at a secondsignal rate inherent to the fourth signal;

pre-processing storage means for temporarily storing the fourth signalat the second signal rate and for repeatedly outputting elements of thestored signal as a first signal at a slower first signal rate; and

processing means for processing the first signal at the first rate andfor outputting at the first signal rate a second processed signal havinga third inherent signal rate which is faster than the first signal rate;and

recording means for recording the second signal.

Accordingly, the source means can operate at normal speedintermittently, and the pre-process ing storage means can provide theslow-motion output to the processing means.

In one embodiment, the pre-processing storage means may comprise:

at least two sub-storage means; and

control means for controlling the sub-storage means such that portionsof the fourth signal are stored intermittently and alternately in thesub-storage means, such that, while a portion of the fourtim signal isbeing stored in one of the sub-storage means, a portion of the firstsignal is output from the other, or one of the other, substorage means,and such that the first signal is output continuously.

However, in another embodiment, the pre-processing storage means isoperable:

to store the fourth signal intermittently at storage locations which arecycled continuously in the storage means;

to output the first signal continuously; and

to store the fourth signal such that no portion of the stored signal isoverwritten before it is output as a portion of the first signal.

In either of these embodiments, the processing means can operatecontinuously, and in the latter embodiment, less storage space isrequired.

It will be appreciated that a single apparatus may be provided embodyingboth the first aspect and the second aspect of the tenth invention.

An aim of a third aspect of the tenth invention is to reduce the amountof starting and stopping of the video tape recorders, or other input andoutput devices, and thus reduce wear.

In accordance with this third aspect of the tenth invention, there isprovided a signal processing method for continuously recording in orderon a medium a signal which is supplied in repeated or intermittentportions (each of which may be a field or frame of a digital videosignal) in which the the between the start of one portion and the next:is R times the duration of each portion, comprising:

a first phase in which the portions are recorded on an intermediatemedium in a scrambled order; and

a second phase in which the scrambled portions are recorded on the finalmedium in the originally supplied order;

(1) the first phase comprising the steps of:

(1a) recording a group of the supplied signal portions on theintermediate medium with a spacing of R portions between the start ofone portion and the next and with the first of the portions beingrecorded at a predetermined starting location on the intermediatemedium; and

(1b) repeating the recording step a number of times until all of thesignal portions have been recorded and with the predetermined startinglocation on the intermediate medium of each succeeding group beingbetween the recording locations of a pair of recorded portions of apreceding group and offset so that no recorded portion is overrecorded;and

(2) the second phase comprising the steps of:

(2a) playing back the intermediate medium and temporarily storing everyRth portion starting at a predetermined location on the intermediatemedium until C portions have been stored;

(2b) recording the C stored portions continuously on the final mediumstarting at a predetermined location on the final medium, while theintermediate medium is still advancing;

(2c) playing back the intermediate medium and temporarily storing everyRth portion until a further C portions have been stored, while the finalmedium is still advancing;

(2d) recording the C stored portions continuously on the final medium,while the intermediate medium is still advancing;

(2e) repeating steps `2c` and `2d` until the end of the recording on theintermediate medium is reached;

(2f) repeating steps `2a` to `2e` until all of the portions have beenrecorded on the final medium and with different predetermined startinglocations such that the portions are recorded in order on the finalmedium.

Preferably, the offset between successive predetermined startinglocations in steps `1a` and `1b` is C(R+1).

Preferably, the offset between the locations on the intermediate andfinal media in steps `2b` to `2f` is C(PR+R-P), where P is the number oftimes the intermediate medium has been played back, starting counting atzero.

Preferably, the value of C and R satisfy the conditions that C modulo Ris non-zero, and C modulo R and R do not have a common factor.

An embodiment of the tenth invention is described particularly withreference to FIGS. 71 to 74 of the accompanying drawings.

Description of the Prior Art Relevant to the Eleventh Invention

Patent Application GB2231228A describes an arrangement for converting 60field/s 2:1 interlace HDVS to 24 Hz 1:1 film format. In thatarrangement, the output frames are either temporally aligned withrespective input frames or temporally offset by one half of an inputfield period (1/120s). In the case of temporal alignment, an outputframe is based upon a respective progressive format frame developed froman input field and temporally adjacent input fields, whereas in the caseof a temporal offset, each pixel in the output frame is 1/2:1/2interpolated between pixels or pixel patches in preceding and succeedingprogressive format frames, with spatial offset between the pixels orpatches in the source frames and the pixel in the output frame beingdependent upon a motion vector which is developed for that pixel.

In the case where there is no temporal offset and a pixel at location(x,y) in the output frame has a motion vector (m,n), this pixel isderived from the pixel at location (x,y) or a patch centred on (x,y) inan input frame 1 to an interpolator, and the motion vector and thecontent of an input frame 2 are not employed. Also, in the case wherethere is a half field period temporal offset and an even motion vector(m,n) for an output pixel at location (x,y), the value of this pixel isderived by equal interpolation between the pixel at (or patch centredon) location (x,y)-1/2(m,n) in input frame 1 and the pixel/patch atlocation (x,y)+1/2(m,n) in input frame 2. However, in the case whereeither of the components of the motion vector (m,n) is an odd integer,the required pixels or patches in the input frames 1 and 2 will be atlocations which are half-way between actual pixel positions in the inputframes.

In order to acquire the required pixel values from the input frames, atwo-dimensional patch may be used around the required pixel location,and there will therefore be an offset of (0,0), (1/2,0), (0, 1/2), or(1/2,1/2) between the centre pixel of the patch and the pixel locationdetermined by the interpolator depending upon whether neither, one orthe other, or each of the motion vector components is odd. To determinethe value of the required pixel, spatial interpolation coefficients areapplied to the pixels in the patch, and the sets of coefficients may bechosen to be slightly different for the four possible offsets, althoughthe coefficients for the cases of offset (1/2,0) and (0, 1/2) may besymmetrical about the x=y diagonal of the patch.

A problem which can arise with such an arrangement is that the magnituderesponses for the four different sets of spatial interpolationcoefficients can be different and produce modulation of the picturedetail as the different responses are cycled.

OBJECT AND SUMMARY OF THE ELEVENTH INVENTION

One aspect of the eleventh invention aims to deal with this problem ofundesirable picture detail modulation.

In accordance with said one aspect of the eleventh invention, there isprovided a method of forming an output image by motion compensatedinterpolation between a pair of input images, comprising the steps, foreach pixel location in the output image, of:

producing a respective motion vector;

determining respective source locations in the input images independence upon the respective motion vector; and

calculating the value of the respective pixel in the output image fromrespective values derived for the pixels at the source locations in theinput images;

wherein the source locations are displaced from the location in theoutput image by amounts proportional to the respective motion vectorplus a constant amount. In an ideal case, the constant amount is onequarter of a pixel location spacing in each dimension and this willresult in the possible offsets being (+or -1/4, or -1/4).

Therefore, all of the possible offsets are rotationally symmetrically,and thus picture detail modulation can be reduced or obviated. Thismethod is of particular advantage when the output image is to beinterpolated half-way between the input images, and in this case theconstants of proportionally are preferably chosen to be equal inmagnitude and opposite in sign for the respective two source locations.

In the arrangement described in GB2231228A, with conversion from 60field/s 2:1 interlace HDVS to 24 Hz 1:1 film format, every other outputframe from the interpolator 48 is produced from one of the progressiveframes input to the interpolator, and the alternate output frames areproduced by motion compensation between two progressive frames input tothe interpolator. This can result in (a) perspective changes not beingsatisfactorily merged, (b) alias effects when the progressive scanconversion fails due to noise, and (c) noise level modulation when theinput image is noisy. As regards point (b), when progressive scanconversion fails due to noise, the progressive scan frames are producedby intrafield interpolation. Of such a frame is directly output by theinterpolator, stationary images would appear heavily aliased.

A second aspect of the eleventh invention aims to deal with thisproblem, an in accordance with said second aspect, there is provided amethod of forming a series of output images from a series of inputimages in which some of the output images are temporally aligned withrespective input images and some of the output images are temporallyoffset from the input images, comprising the steps of:

producing a respective motion vector for each pixel location in eachoutput image;

in the case of a temporally offset output image, determining, for eachpixel location therein, respective source locations in the temporallypreceding and temporally succeeding input images which are displacedfrom the pixel location in the output image by amounts dependent uponthe respective motion vector and the temporal offset;

in the case of a temporally aligned output image, determining, for eachpixel location therein, a respective source location in the temporallyaligned input image, and a respective source location in the precedingor succeeding input image which is displaced from the pixel location inthe output image hy an amount dependent upon the respective motionvector; and

calculating for each pixel location in each output image the value ofthe pixel from respective values derived for the pixels at the twosource locations in the respective input images.

By producing each pixel in the output frame from two input frames,whether or not there is a temporal offset between the output: frame andthe input frames, alias is removed, because the frame 2 alias willalways be in antiphase to the frame 1 alias as long as the interfieldmotion is an exact multiple of lines. As synthesised lines are mixedwith non-synthesised lines in this scheme, an improved vertical responseis also produced. A further advantage is that if there is noise in theinput image, the noise will not be modulated, unlike the case whereevery other output frame is derived from only one of the input frames.

It will be appreciated that the methods of both aspects of the inventionmay be combined in the same method.

In the case of either method, or the combined method, the values derivedfor the pixels at the source locations are preferably derived by spatialinterpolation between a set of pixel values around the source location.

An embodiment of the eleventh invention is described particularly withreference to FIGS. 61 to 65 of the accompanying drawings.

Description of the Prior Art Relevant to the Twelfth Invention

The aforementioned patent application GB2231228A describes a method ofmotion compensated interpolation of an output image between a pair ofinput images comprises the steps of:

developing at least one local motion vector for each pixel in the outputimage area indicative of estimated motion of that pixel in the outputimage;

determining, as at least one global motion vector, at least the mostfrequently occurring local motion vector for the whole output imagearea;

selecting, for each pixel, an output motion vector from the respectivelocal motion vector or vectors, and the global motion vector or vectors;and

determining, for each pixel in the output image, the value thereof byinterpolation between pixels in the input images displaced from thelocation of the pixel in the output image by amounts determined by theselected output motion vector.

In the system described in that application only a small number ofmotion vectors can be tested on a pixel-by-pixel basis. For optimumoperation of the system it is important that the best vectors arepreselected for testing by a motion vector selector. Techniques usingglobal motion vectors only have proved to be good for many types ofpicture and techniques using only locally derived motion vectors haveproved good for certain material. Neither is good for all material.

OBJECT AND SUMMARY OF THE TWELFTH INVENTION

In accordance with the twelfth invention, there is provided a method ofmotion compensated interpolation of an output image between a pair ofinput images, comprising the steps of:

developing at least one local motion vector for each pixel in the outputimage area indicative of estimated motion of that pixel in the outputimage;

determining, as at least one global motion vector, at least the mostfrequently occurring local motion vector for the whole output imagearea;

determining, for all least one intermeciiate portion of the output imagearea, as at least one respective intermediate motion vector, at leastthe most frequently occurring local motion vector for pixels in thatintermediate area portion;

selecting, for each pixel, an output motion vector from the respectivelocal motion vector or vectors, the intermediate motion vector orvectors for at least one intermediate area related to the position ofthat pixel, and the global motion vector or vectors; and

determining, for each pixel in the output image, the value thereof byinterpolation between pixels in the input images displaced from thelocation of the pixel in the output image by amounts determined by theselected output motion vector.

This technique therefore combines good points from both of theapproaches described in the earlier application.

In one example the, or at least some of the, intermediate area portionsare predefined, and there may be a plurality of such intermediate areaportions. In this case, at least some of the intermediate area portionsmay be in a contiguous array, and/or at least some of the intermediatearea portions may be in an overlapping array, and there may be apluralily of such arrays.

In an example of the method at least one of such intermediate areaportions is determined to be related to such a pixel position if thepixel position is within that intermediate area portion, or if the pixelposition is within a respective area of application larger than andincluding that intermediate area portion, or if the pixel position iswithin a respective area of application smaller than and within thatintermediate area portion.

An embodiment of the twelfth invention is described particularly withreference to FIGS. 75 to 79 of the accompanying drawings.

Other objects, features and advantages of the present twelve inventionswill become apparent upon consideration of the following detaileddescription of preferred embodiments thereof, especially when consideredwith the accompanying drawings in which like reference numerals areemployed to designate the same or similar components in the differentfigures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a previously proposed apparatus for videosignal to photographic film conversion;

FIG. 2 is a block diagram of part of an embodiment of apparatus forvideo signal to photographic film conversion according to the presentinvention;

FIG. 3 is a block diagram of another part of the embodiment;

FIG. 4 is a more detailed block diagram of part of the embodiment;

FIG. 5 shows diagramatically progressive scan conversion;

FIGS. 6 to 9 show diagramatically sequences of lines in sequences offields explaining progressive scan conversion;

FIG. 10 is a block diagram showing the steps in motion adaptiveprogressive scan conversion;

FIG. 11 shows diagrammatically progressive scanning, in particular therequired estimate and difference value between successive fields;

FIGS. 12 and 13 are diagrams used in explaining the technique of FIG. 11in more detail, FIG. 12 showing a progressive scan normalizing functionand FIG. 13 showing a progress ive scan nonlinear function;

FIG. 14 shows diagrammatically the creation of pixels in missing linesin progressive scan conversion;

FIGS. 15A, 15B, and 16 show diagrammatically search blocks and searchareas, and the relationships therebetween;

FIGS. 17 shows a correlation surface;

FIGS. 18, 19A and 19B show diagrammatically how a search block is grown;

FIG. 20 shows the areas of a frame in which search block matching is notpossible;

FIGS. 21A and 21B show diagrammatically a moving object straddling threesearch blocks;

FIGS. 22 to 24 show three resulting correlation surfaces, respectively;

FIGS. 25 and 26 show further examples of correlation surfaces, used indescribing a threshold test;

FIGS. 27 and 28 show still further examples of correlation surfaces,used in describing a rings test;

FIGS. 29 shows diagrammatically how the direction in which a searchblock is to grow is determined;

FIGS. 30A and 30B show diagrammatically how a correlation surface isweighted;

FIG. 31 shows the relationship between sample blocks and search blocks,and a frame of video;

FIG. 32 shows motion vector regions in a frame of video;

FIGS. 33 to 35 show diagrams used in explaining motion vector reductionin respective regions of a frame of video;

FIGS. 36 and 37 show diagrammatically a first stage in motion vectorselection;

FIGS. 38 and 39 show diagrammatically how a threshold is establishedduring the motion vector selection;

FIG. 40 shows diagrammatically a second stage in motion vectorselection;

FIGS. 41 to 47 show arrays of pixels with associated motion vectors,used in explaining motion vector post-processing;

FIG. 48 shows diagrammatically the operation of an interpolator.

FIG. 49 is a diagram illustrating the correlation between frames of a 24Hz 1:1 format signal and a 60 field/s 3232 pulldown format signal;

FIG. 50 is a diagram illustrating signal conversion from 60 field/s 2:1interlace format to 60 field/s 3232 pulldown format;

FIG. 51 shows a modification to part of FIG. 50;

FIG. 52 is a diagram illustrating a basic correlation between frames ofa 30 Hz 1:1 format signal and fields of a 60 field/s 2:1 interlaceformat signal with interpolation of alternate output fields only;

FIGS. 53A-53C-3 show a modification to FIG. 52 with interpolation of alloutput fields;

FIG. 54 is a diagram illustrating signal conversion from 30 Hz 1:1format to 60 field/s 2:1 interlace format;

FIGS. 55A and 55B are diagrams illustrating a correlation between framesof a 24 Hz 1:1 format signal and fields of a 60 field/s 2:1 interlaceformat signal;

FIG. 56 is a diagram illustrating signal conversion from 24 Hz 1:1format to 60 field/s 2:1 interlace format;

FIG. 57 is a diagram illustrating the source fields of a 60 field/s 2:1interlace format signal used to produce each frame of a 30 Hz 1:1 formatsignal with motion adaptive progressive scan conversion only;

FIG. 58 is a diagram illustrating the signal conversion of FIG. 57;

FIG. 59 shows a modification to FIG. 57 with motion compensationinterpolation also;

FIG. 60 is a diagram illustrating the signal conversion of FIG. 59;

FIG. 61A-1 to 61C-3 show three examples of the relation between a pixelin an output frame and the respective source pixels in two input framesof the motion compensation interpolator of FIG. 48;

FIGS. 62A to 62D show the different four possible offsets between thelocation of a required pixel in a input frame shown in FIGS. 61A-1 to61C-3 and the actual pixel positions in the input frame;

FIGS. 63A-1 to 63C-3 are similar to FIGS. 61A-1 to 61C-3 respectively,but showing the case where a global offset: of (-1/4 pixel , -1/4 pixel)is applied;

FIGS. 64A to 64D are similar to FIGS. 62A to 62D, respectively, butshowing the offset of FIGS. 63A-1 to 63C-3;

FIGS. 65A-65C are similar to a combinaLion of FIGS. 63A and 64A, butshowing how a pixel in an output frame can be derived from pixels inboth input frames, even when there is no temporal offset;

FIG. 66 illustrates an overall system primarily for transfer from 24 Hz1:1 film to 24 Hz 1:1 film and allowing post production integration with60 field/s 2:1 inLerlaced format material;

FIG. 67 illustrates an overall system primarily for transfer from 24 Hz1:1 film to 60 field/s 2:1 interlace HDVS and allowing post productionintegration with 60 field/s 2:1 interlaced format material;

FIG. 68 illustrates an overall system primarily for transfer from 30 Hz1:1 film to 30 Hz 1:1 film and allowing post production integration with60 field/s 2:1 interlaced format: material and 30 Hz 1:1 formalmaterial;

FIG. 69 illustrates an overal 1 system primarily for transfer from 30 Hz1:1 film and 60 Hz 1:1 film to 60 field/s 2:1 interlace HDVS andallowing post production integration with 60 field/s 2:1 interlacedformat material.

FIG. 70 illustrates an overall system primarily for transfer from 30 Hz1:1 to 24 Hz 1:1 or 30 Hz 1:1 film and allowing post productionintegration with 30 Hz 1:1 video or 60 Hz 2:1 video;

FIG. 71 shows a modification to FIG. 2 to increase the conversion rate;

FIG. 72 shows a modification to FIG. 2 to enable conversion to takeplace in two phases requiring less starting and stopping of the videotape recorders;

FIG. 73 shows a modification to FIG. 2 which obviates the need for aslow-motion source video tape recorder;

FIG. 74 shows a modification to FIG. 73 which also increases theconversion rate;

FIGS. 75 to 78 illustrates how an image area may be sub-divided intointermediate areas for determining and applying intermediate motionvectors; and

FIG. 79 illustrates schematically how local, global and variousintermediate motion vectors are made available for selection.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiment of apparatus for video signal to photographic filmconversion to be described is particularly intended for use in theconversion of a high definition video signal (HDVS) having 1125 linesper frame, 60 fields per second, to 24 frames per second 35 mm film.However, it will be understood that the invention is not limited in thisrespect, and that it can readily be adapted to effect conversion fromother input video signals.

The apparatus can conveniently be considered in two parts; the firstpart, shown in FIG. 2, effects the conversion of the input HDVS to aprogressive scan digital video signal corresponding to 24 frames persecond which is recorded on a VTR; and the second part, shown in FIG. 3,reproduces the recorded video signal and transfers it to photographicfilm.

The part of the apparatus shown in FIG. 2 comprises a high definitiondigital VTR 11, a television standards converter 12, a frame recorder 13which can record up to say one second of video signal, a second highdefinition digital VTR 14, a system controller 15 having associated withit a tracker ball control 16, a keyboard 17 and a graphics display 18,and television monitors 19 and 20, interconnected as shown, andoperating as will be described below.

The second part of the apparatus, shown in FIG. 3, comprises a highdefinition digital VTR 31, a digital interface (I/F) unit 32, a gammacorrector 33, a digital-to-analogue converter 34, an electron beamrecorder 35, a television monitor 36 and a switch 37, interconnected asshown, and operating as will be described below.

Referring again to FIG. 2, the video signal connect. ions D are digitalconnections, that is carrying Y, U/V s i gnals, and the video signalconnections A are analogue connections carrying R, G, B signals. Theinput video signal which is to be transferred to film, and which mayhave been derived from a high definition video camera, is recorded on amagnetic tape reproduced by the digital VTR 11. The digital VTR 11 iscapable of reproducing the recorded video signal at: 1/8 speed, as thisis a convenient speed of operation for the subsequence circuitry, and inparticular the standards converter 12. The elements 11 to 14, 19 and 20are under control of the system controller 15, the system controller 15being in turn controllable by inputs from the tracker bail control 16and the keyboard 17, and having associated with it the graphics display18 on which is displayed information relating to the progress of theconversion.

A portion of the input HDVS is reproduced from the digital VTR 11 andsupplied to the standards converter 12. This operates, as described indetail below, to derive from the input video signal, which is a 60fields per second interlace scanned video signal, firstly, a motionadapted progressive scan digital video signal at 60 frames per second,and then from this the required motion compensated progressive scandigital video signal corresponding to 24 frames per second, but notnecessarily at that rate. This video signal is recorded by the digitalVTR 14, and if the digital VTR 14 is capable of recording in slowmotion, that is at the reproduction rate of the digital VTR 11, then intheory the frame recorder 13 is not required. In practice, however, theframe recorder 13 may in any case be a useful addition to the apparatus,as it more readily permits intermittent operation to be effected. Suchintermittent operation is generally required for video signal to filmconversion, because of the need to check at frequent intervals that theconversion is proceeding satisfactorily. Thus depending on the contentof the video signal to be converted, adjustment of the parameters, inparticular those of the standards converter 12, need to be made, and theresults evaluated before proceeding. The monitors 19 and 20 are providedas further means for checking the video signal at respective points inthe apparatus.

In the second part of the apparatus, shown in FIG. 3, the motioncompensated progressive scan digital video signal recorded by thedigital VTR 14 (FIG. 2) is reproduced by the digital VTR 31 and passedby way of the digital I/F unit 32 to the gamma corrector 33, the purposeof which is to match the gamma characteristics of the video signal tothe gamma characteristics of the film being used. The separated operation permitted by the recording of the motion compensated progressivescan digital video signal by the digital VTR 14 (FIG. 2), for subsequentreproduction by the digital VTR 31, enables the gamma correction to beset accurately by the gamma corrector 33, because intermittent andrepeated operation is possible so that various different mappings of thegenerally non-linear gamma characteristics of the video signal from thedigital VTR 31 to the generally linear gamma characteristics of the filmcan be tested. This gamma setting may, for example, involve the use of astep wedge. The gamma corrected digital video signal is then convertedto an analogue signal by the digital-to-analogue converter 34 andsupplied to the electron beam recorder 35 to be recorded on photographicfilm. This recording may, for example, be in the form of threemonochrome frames for each frame of the video signal, the three framescorresponding respectively to red, green and blue. The furthertelevision monitor 36 can be selectively connected by way of the switch37 to the output of the digital VTR 31 or to the output of thedigital-to-analogue converter 34, or alternatively of course twoseparate television monitors can be provided.

The characteristics of the apparatus are such that it produces sharp,clear pictures with good motion portr-ayal on the film, and inparticular it produces pictures without motion blur and withoutintroducing any additional judder components. Moreover, the separatedoperation permitted by the recording of the motion compensatedprogressive scan digital video signal on the digital VTR 14, in turnpermits easy and frequent checking of the parameters of the apparatus,to ensure the quality of the pictures obtained on the film. Iterativeoperation is perfectly possible, so that the results can rapidly beevaluated and conversion repealed with any flaws corrected by adjustmentof the parameters. To obtain higher speed operation, it is of coursepossible for the first part of the apparatus, that is the part shown inFIG. 2 to be replicated a number of times, to provide additional inputsto the digital VTR 31, so permitting a more intensive use of the part ofthe apparatus shown in FIG. 3, and hence a higher overall conversionspeed.

FIG. 4 is a block diagram of the standards converter 12 which will nowbe described in more detail. The standards converter 12 comprises aninput terminal 41 to which an input video signal is supplied. The inputterminal is connected to a progressive scan converter 42 in which theinput video fields are converted into video frames which are suppl ledto a direct block marcher 43 wherein correlation surfaces are created.These correlation surfaces are analysed by a motion vector estimator 44,which derives and supplies motion vectors to a motion vector reducer 45,wherein the number of motion vectors for each pixel is reduced, beforethey are supplied to a motion vector selector 46, which also receives anoutput from the progressive scan converter 42. Any irregularity in theselection of the motion vectors by the motion vector selector 46 isremoved by a motion vector post processor 47, from which the processedmotion vectors are supplied to and control an interpolator 48 which alsoreceives an input from the progressive scan converter 42. The output ofthe interpolator 48, which is a standards-converted andmotion-compensated video signal is supplied to an output terminal 49.Each part of the standards converter 12 and the operation thereof willbe described in more detail below.

The progressive scan converter 42 produces output frames at the samerate as the input fields. Thus, referring to FIG. 5 which shows asequence of consecutive lines in a sequence of consecutive fields, thecrosses representing lines present in the input fields and the squaresrepresenting interpolated lines, each output frame will contain twicethe number of lines as an input field, the lines alternating betweenlines from the input video signal and lines which have been interpolatedby one of the methods to be described below. The interpolated lines canbe regarded as an interpolated field of the opposite polarity to theinput field, but in the same temporal position.

Progressive scan conversion is preferably carried out, for two mainreasons; firstly, to make the following direct block matching processeasier, and secondly in consideration of the final output video format.These two reasons will now be considered in more detail.

Direct block matching is used to obtain an accurate estimation of thehorizontal and vertical motion between two successive video fields, asdescribed in more detail below. However, due to the interlaced structureof the video signal on which direct block matching is performed,problems can arise.

Consider the image represented by FIG. 6, which indicates a sequence ofsuccessive lines in a sequence of successive fields, the open squaresrepresenting white pixels, the black squares representing black pixels,and the hatched squares representing grey pixels. This, therefore,represents a static picture with a high vertical frequency componentwhich in a HDVS would be 1125/3 cycles per picture height. As this imagehas been sampled by the usual interlace scanning procedure, each fieldappears to contain a static vertical frequency luminance component Y of1125/6 cph, as indicated in FIG. 7. However, the frequency components ineach field are seen to be in antiphase. Attempts to perform direct blockmatching between these two fields will lead to a number of differentvalues for the vertical motion component, all of which are incorrect.This is indicated in FIG. 8, in which the abbreviation LPF means linesper field. From FIG. 8 it is cIear that direct block matching will not.give the correct answer for the vertical motion component, whichcomponent should in fact be zero. This is because the direct blockmatching is in fact tracking the alias component of tire video signalrather than the actual motion.

Consider now FIG. 9, which depicts the same static image as FIG. 6,except that now each input field has been progressive scan converted toform a frame, the triangles representing interpolated pixels. It can beseen that each frame now contains the same static vertical frequencycomponent as the original input fields, that is 1125/3 cph. Thus, elifeet block matching between two successive frames can now give thecorrect value for the vertical motion, that is, zero, and the trackingof the vertical alias has been avoided. Moreover, there is the pointthat direct block matching on progressive scan converted frames willresult in a more accurate vertical motion estimate, because the directblock matching is being performed on frames which have twice the numberof lines.

Concerning consideration of the final output video format, in the caseof the present embodiment, the converteel video is supplied via tape toan electron beam recorder, and needs to consist of frames correspondingto the motion picture film rate of 24 frames per second. For thisreason, therefore, the production of progressive scan converted framesis necessary, and moreover the progressive scan converted frames canalso be used as a fall-back in the case where motion compensatedstandards conversion is deemed to be producing unacceptable results, forexample, where the motion is too diverse to be analysed satisfactorily.In that case the use of the nearest progressive scan converted frame asthe required output frame can produce reasonably acceptable results.

Progressive scan conversion can be carried out in a number of ways, suchas by previous field replacement, median filtering in which threespatally consecutive lines are examined (temporally these three lineswill come from two consecutive fields), or a motion compensatedtechnique which utilizes multi-gradient motion detection followed bymulti-direction linear interpolation. However, in the present embodimentthe preferred method is motion adaptive progressive scan conversion, thesteps of which are indicated in the block diagram of FIG. 10. Theconcept is to use inter-field interpolation in wholly static pictureareas to retain as much vertical information as possible, and to useintra-field interpolation when significant motion is present. This alsoaids smooth portrayal of motion. In scenes where the motion is somewherebetween these two extremes, an estimate of the local motion present inthe picture is made, and this is then used to mix together differentproportions of inter- and intra-field interpolation.

In more detail, the modulus of the frame difference between previous andnext fields is first generated, this being indicated in FIG. 11. Togenerate the required estimates, the modulus inter-frame differencearray from the previous and the next fields is generated at each point:##EQU1## where:

.increment._(U) is the unnormalized modulus difference array, and

Y is the luminarice array corresponding to the 3D picture.

The modulus of difference is then normalized to adjust for thesignificance of changes in lower luminance areas: ##EQU2## where:

.increment._(N) is the normalized modulus difference array

Y is the inter-frame average luminanee value

Y (pixel, current line)=(Y (pixel, current line, previousfield)+Y(pixel, current line, next field) )/2, and

F(Y) (the normalizing function) is derived as indicated in FIG. 12.

The difference array .increment. is then vertically filtered togetherwith the previous field difference by a three-tap filter (examples ofcoefficients are a quarter, a half, a quarter or zero, unity, zero) toreduce vertical alias problems, and in particular to minimize theproblems encountered with temporal alias. Thus: ##EQU3## where:

.increment._(F) is the filtered normalized difference array, and

C₁ and C₂ are filter coefficients, and 2C₁ +C₂ =1 so that unity dc gainis maintained.

A vertical and horizontal intra-field filter of up t.o five Laps byfifteen taps is then used to smooth the difference values within thecurrent field. In practice, a filter of three taps by three taps issatisfactory. Finally, in order to produce the actual motion estimation,a non-linear mapping function is applied using a function to provide themotion estimate (ME):

ME (pixel, current line)=γ(spatially filtered .increment._(F) (pixel,current line) )

The non-linear function γ derived as shown in FIG. 13, the staticpicture ME is zero, for full motion ME is one, and for intermediatemotions a controlled transition occurs.

To produce an interpolated pixel, the pixels in the missing line arecreated by taking proportions of the surrounding lines as indicated inFIG. 14. The motion estimate ME is then applied to the intraframeinterpolated value (generated from a two, four, six or preferably eighttap filter), and 1-ME is applied to the inter-field average (oralternatively to a more complex interpolated value), and these aresummed to derive the progressive scan pixel estimate: ##EQU4## where:

C₀, C₁, C₂ and C₃ are the intra-frame filter coefficients, and 2(C₀ +C₁+C₂ +C₃)=1 so that unity dc gain is maintained.

This method of progressive scan conversion is found to produce highquality frames from input fields, in particular because a moving objectcan be isolated and interpolated in a different manner to a stationarybackground.

Referring back to FIG. 4, the frames of video derived by the progressivescan converter 42 are used to derive motion vectors. Time estimation ofmotion vectors consists of two steps. Firstly, correlation surfaces aregenerated by correlating search blocks from consecutive frames. Then,having obtained these correlation surfaces, they have to be examined todetermine the position or positions at which correlation is best.Several different methods of obtaining a correlation surface exist, thetwo main methods being phase correlation and direct block matching.There are, however, a number of problems associated with the use ofphase correlation, these being very briefly problems relating to thetransform mechanism, the windowing function, the block size and thevariable quality of the contour of the surface produced. In the presentembodiment, therefore, direct block matching is preferred.

The direct block marcher 43 operates as follows. Two blocks,respectively comprising a rectangular array of pixels from consecutiveframes of the progressive scan converted video signal are correlated toproduce a correlation surface from which a motion vector is derived.

Referring to FIG. 15, firstly a small block called a search block ofsize 32 pixels by 23 lines is taken from a frame as shown in FIG. 15.Then a larger block called a search area of size 128 pixels by 69 linesis taken from the next frame. The search block (SB) is then placed ineach possible position in the search area (SA) as shown in FIG. 16, andfor each location the sum of the absolute difference of pixel luminaneelevels between the two blocks is calculated. This value is then used asthe height of the correlation surface at the point at which it wasderived. It can then be used in conjunction with other similarly derivedvalues for each possible location of the search block in the search areato obtain a correlation surface, an example of which is shown in FIG.17. For clarity the surface is shown inverted, and as it is in fact theminimum that is required, the required point in FIG. 17 is the mainpeak.

The size of the search block is selected by examining the minimum sizeof an object that may require motion compensation. For PAL 625 lines perframe, 50 fields per second signals a search block of 16 pixels by 8lines has been found suitable for tracking a small object withoutallowing any surrounding information not within the object, but stillwithin the search block, to affect the tracking of the object. Thisapproach bas therefore been adopted in the present embodiment, butmodified to take account of the different numbers of active pixels perline, active lines per frame, and aspect ratio of a HDVS as comparedwith PAL 625/50. The comparative figures, the HDVS being put first, areas follows; 1920 (720) active pixels per line, 1035 (575) active linesper frame, 3:5.33 (3:4) aspect ratio.

It should be added that there is an argument for using a larger searchblock, since this means that a large object can be tracked. On the otherhand, there exists an argument for using a smaller search block, toprevent a small object being over-shadowed by the effect of a largeobject or background area. Also, however, there is the advantage thatwith small search blocks there is no requirement for the derivation ofmore than one motion vector from each of them. Because having a singlemotion vector is so much easier than having more than one, the presentembodiment starts with a small search block as described above, and thencauses the search block to grow into a bigger search block if nosatisfactory result has been obtained. This then encompasses theadvantages of both a small and a large search block. The criteria for asatisfactory result is set by the motion vector estimator 44 (FIG. 4)referred to in more detail below and which determines the motion vectorfrom a given correlation surface.

This technique of causing the search block to grow is not onlyadvantageous for tracking large objects. It can also help to track themovement of an object hay ing the shape of a regular pattern of aperiodic nature. Thus, consider FIG. 18 where a search block A willmatch up with the search area B at locations V1, V2 and V3, with each ofthem giving a seemingly correct measure of motion. In this case,however, the motion vector estimation, that is the process that actuallyanalyses the correlation surface, will show that good correlation occursin three locations which are collinear. The search block will thereforebe caused to grow horizontally until it is three times its originalwidth, this being the direction in which multiple correlation occurredin this case. The search area will also be correspondingly horizontallyenlarged. As shown in FIG. 19, with the enlarged search block 3A, thereis only a single correlation point, which correctly relates to themotion of the object.

In this particular case the search block and the search area both haveto grow horizontally, because the direction of multiple correlation ishorizontal. It is equally possible, however, for the search block andthe search area to grow vertically, or indeed in both directions, if thecorrelation surface suggests it.

It should be noted that block matching cannot be applied to all thesearch blocks in the frame, because in the border area there is notenough room from which a search area can be drawn. Thus, block matchingcannot be effected in the border area of the frame shown hatched in FIG.20. This problem is dealt with by the motion vector reducer 45 (FIG. 4)described in more detail below, which attempts to supply search blocksin this hatched area with appropriate motion vectors.

From the correlation surface (FIG. 17) generated for each search blockin a frame the motion vector estimator 44 (FIG. 4) deduces the likelyinter-frame motion between the search block and its corresponding searcharea. It should again be mentioned that for clarity all diagrams ofcorrelation surfaces are shown inverted, that is, such that a minimum isshown as a peak.

The motion vector estimator 44 (FIG. 4) uses motion vector estimationalgorithms to detect the minimum point on each correlation surface. Thisrepresents the point of maximum correlation between the search block andthe search area, and hence indicates the probable motion between them.The displacement of this minimum on the correlation surface with respectto the origin, in this case the centre of the surface, is a directmeasurement, in terms of pixels per frame, of the motion. For thesimplest case, where the correlation surface contains a single, distinctminimum, the detection of the minimum point on the correlation surfaceis sufficient to determine accurately the motion between the searchblock and the search area. As previously mentioned, the use of smallsearch blocks improves the detection of motion and the accuracy ofmotion estimation, but unfortunately small single search blocks areunable to detect motion in a number of circumstances which will now bedescribed.

FIG. 21 shows an object with motion vectors (5, 0) straddling threesearch blocks 1A, 2A and 3A in a frame (t). When the search blocks 1Aand 3A are correlated with respective search areas (1B and 3B) in thenext frame (t+1) a correlation surface shown in FIG. 22 results showinga minimum at (5, 0). (This assumes a noiseless video source.) However,when the search block 2A is correlated with its respective search area2B, the correlation surface shown in FIG. 23 is produced, in which thesearch block 2A correlates with the search area 2B at every point in they-axis direction. There is therefore no single minimum in thecorrelation surface, and hence the motion between the search block 2Aand the search area 2B cannot be determined.

However, now consider the situation if the search block 2A is grown suchthat it encompasses all three of the original search blocks 1A, 2A and3A. When the grown search block 2A is correlated with a search areacovering the original search areas 1B, 2B and 3B, the resultingcorrelation surface is as shown in FIG. 24. This shows a single minimumat (5, 0) indicating the correct motion of the original search block 2A.This example illustrates the need for some unique feature in the sourcevideo, in order accurately to detect motion. Thus, the search blocks 1Aand 3A both had unique vertical and horizontal features, that is theedges of the object, and hence motion could be determined. In contrast,the search block 2A had a unique vertical feature, but no uniquehorizontal feature, and hence horizontal motion could not be determined.However, by growing the search block until it encompasses a uniquefeature both horizontally and vertically, the complete motion for thatsearch block can be determined. Moreover, it can be shown that growingthe search block is beneficial when noise in the source video isconsidered.

A further example will now be considered with reference to FIG. 25. Thisshows a correlation surface for a search block where the motion vectoris (5, 3). However, due to the numerous other correlations which havetaken place between the search block and the search area, the truemotion is difficult to detect. An example of source video which mightproduce such a correlation surface would be a low contrast tree movingwith the wind. It is now assumed that the search block and the searcharea are grown. The growing can take place in the horizontal direction,as in the previous example, or in the vertical direction, or iu bothdirections. Assuming that the neighbouring search blocks have the samemotion, the mean effect on the resulting correlation surface will be toincrease the magnitude of the minima at (5, 3) by a greater proportionthan the magnitude of the other correlation peaks. This is shown in FIG.26, which indicates that it is then easier to detect the correct motionvector.

The way in which search blocks are grown will now be further consideredwith reference to FIG. 21. Here it was required to grow the area of thesearch block 2A to encompass the areas of the search blocks 1A and 3A,and to produce the resulting correlation surface. In fact, the resultingcorrelation surfaces are produced directly by adding together theelements of the three correlation surfaces corresponding to the searchblocks 1A, 2A and 3A. In effect, if each correlation surface isconsidered as a matrix of point magnitudes, then the correlation surfaceof the enlarged search block 2A is the matrix addition of thecorrelation surface of the original search blocks 2A and 3A.

The area of the search block 2A could also be grown vertically by addingcorrelation surfaces of the search blocks above and below, whilst if thesearch block 2A is to be grown both horizontally and vertically, thenthe four neighbouring diagonal correlation surfaces have to be added aswell. From this it will be seen that the actual process of growing asearch block to encompass neighbouring search blocks is relatively easy,the more difficult process being to decide when growing should takeplace, and which neigbbouring search blocks should be encompassed.Basically, the answer is that the area of the search blocks should begrown until a good minimum or good motion vector is detected. It istherefore necessary to specify when a motion vector can be taken to be agood motion vector, and this can in fact be deduced from the examplesgiven above.

In the example described with reference to FIGS. 21 to 24, it wasnecessary to grow the search block horizontally in order to encompass aunique horizontal feature of the object, and hence obtain a singleminimum. This situation was characterized by a row of identical minimaon the correlation surface of FIG. 23, and a single minimum on thecorrelation surface of FIG. 24. From this the first criteria for a goodminimum can be obtained; a good minimum is the point of smallestmagnitude on the correlation surface for which the difference between itand the magnitude of the next smallest point exceeds a given value. Thisgiven value is known as the threshold value, and hence this test isreferred to herein as the threshold test.

It should be noted that the next smallest point is prevented fromoriginating from within the bounds of a further test, described below,and referred to herein as the rings test. In the case of a rings testemploying three rings, the next smallest point is prevented fromoriginating from a point within three pixels of the point in question.

In the example of FIGS. 21 to 24, the correlation surface of FIG. 23would have failed the threshold test; the search area 2A is thereforegrown and, given a suitable threshold value, the correlation surface ofFIG. 24 will pass the threshold test.

The threshold test can also be used to cause growing in the exampledescribed above with reference to FIGS. 25 and 26. Prior to growing thesearch block, the correct minimum is undetectable, due to the closelysimilar magnitudes of the surrounding points. Given a suitable thresholdvalue, however, the correlation surface will fail the threshold test,and the search block will be grown. As a result, it will then bepossible to detect the minimum among the other spurious points.

It will be seen that the use of a threshold is a subjective test, butthe correct threshold for the correlation surface under test can beselected by normalizing the threshold as a fraction of the range ofmagnitudes within the correlation surface. This also lessens the effectof, for example the contrast of the video source.

The rings test, referred to briefly above, and which is far lesssubjective, will now be further described. The basis of the rings testis to assume that a good minimum (or maximum) will have points ofincreasing (or decreasing) magnitudes surrounding it. FIG. 27illustrates this assumption, showing a minimum at (0, 0) where thesurrounding three rings of points have decreasing mean magnitude. Thisis as opposed to the correlation surface shown in FIG. 28, where therings, and in particular the second inner-most ring, are not ofdecreasing mean magnitude.

In this case the criteria for a good minimum as defined by the ringstest, is that the average slope is monotonic. Therefore for apre-defined number of rings of points surrounding the minimum inquestion, the mean magnitude of each ring when moving from the innermostring outwards, must be greater than that of the previous ring. Returningagain to the example described with reference to FIGS. 21 to 24, it willbe seen from FIGS. 23 and 24 that the correlation surface of FIG. 23would have failed the rings rest, but that the correlation surface ofFIG. 24 would have passed the rings test. Since the rings test comparesmean, and not absolute, magnitudes, it is far less subjective than thethreshold test, and indeed the only variable in the rings test is thenumber of rings considered.

Having described the mechanism for growing a search block, it is nownecessary to consider how by examining the shape of the correlationsurface it is possible to determine the most effective direction inwhich the search block should grow.

Referring again to FIG. 23, this correlation surface resulted wherethere was a unique vertical feature, but no unique horizontal feature.This is mirrored in the correlation surface by the minimum runninghorizontally across the correlation surface, due to the multiplecorrelations in this direction. From this it can be deduced that thesearch block should be grown horizontally. Conversely, should a line ofmultiple correlations run vertically, this would indicate the need togrow the search block vertically, whilst a circular collection ofmultiple correlations would indicate a need to grow the search blockboth horizontally and vertically.

Using this criteria, a quantative measure of the shape of thecorrelation surface is required in order to determine in which directionthe search block should be grown. This measure is determined as follows.Firstly, a threshold is determined. Any point on the correlation surfacebelow the threshold is then considered. This threshold, like that usedin the threshold test, is normalized as a fraction of the range ofmagnitudes within the correlation surface. Using this threshold, thepoints on the correlation surface are examined in turn in four specificsequences. In each, the point at which the correlation surface valuefalls below the threshold is noted. These four sequences are illustrateddiagrammatically in FIG. 29 in which the numbers 1, 2, 3 and 4 at thetop, bottom, left and right refer to the four sequences, and the hatchedarea indicates points which fall below the threshold:

Sequence 1

Search from the top of the correlation surface down for a point

A which falls below the threshold.

Sequence 2

Search from the bottom of the correlation surface up for a point

C which falls below the threshold.

Sequence 3

Search from the left of the correlation surface to the right for apoint: D which falls below the threshold.

Sequence 4

Search from the right of the correlation surface to the left for a pointB which falls below the threshold.

The locations of the four resulting points A, B, C and D are used tocalculate the two dimensions X and Y indicated in FIG. 29, thesedimensions X and Y indicating the size of the hatched area containingthe points falling below the threshold value. Hence from the dimensionsX and Y, it can be deduced whether the shape is longer in the x ratherthan the y direction, or vice versa, or whether the shape isapproximately circular. A marginal difference of say ten percent isallowed in deducing the shape, that is, the dimension X must be aminimum of ten percent greater than the dimension Y for the shape to beconsidered to be longer in the x direction. Similarly for the ydirection. If the dimensions X and Y are within ten percent of eachother, then the shape is considered to be circular, and the search blockis grown in both directions. In the example of FIG. 29 the dimension Xis greater than the dimension Y, and hence the search block is grown inthe x or horizontal direction.

The growing of the search block continues until one or more growthlimitations is reached. These limitations are: that the minimum in thecorrelation surface passes both the threshold test and the rings test;that the edge of the video frame is reached; or that the search blockhas already been grown a predetermined number of times horizontally andvertically. This last limitation is hardware dependent. That is to say,it is limited by the amount of processing that can be done in theavailable the. In one specific embodiment of apparatus according to thepresent invention, this limit was set at twice horizontally and oncevertically.

If the minimum in the correlation surface passes both the threshold restand the rings test, then it is assumed that a good motion vector hasbeen determined, and can be passed to the motion vector reducer 45 (FIG.4). However, if the edge of the frame is reached or the search block hasalready been grown a predetermined number of times both horizontally andvertically, then it is assumed that a good motion vector has not beendetermined for that particular search block, and instead of attemptingto determine a good motion vector, the best available motion vector isdetermined by weighting.

The correlation surface is weighted such that the selection of the bestavailable motion vector is weighted towards the stationary, that is thecentre, motion vector. This is for two reasons, firstly, if the searchblock, even after growing, is part of a large plain area of sourcevideo, it will not be possible to detect a good motion vector. towever,since the source video is of a plain area, a stationary motion vectorwill lead to the correct results in the suhsequent processing. Secondly,weighting is designed to reduce the possibility of a seriously wrongmotion vector being passed to the motion vector reducer 45 (FIG. 4).This is done because it is assumed that when a good motion vector cannotbe determined, a small incorrect motion vector is preferable to a largeincorrect motion vector.

FIG. 30 shows an example of how the weighting function can be applied tothe correlation surface. In this example, the weight applied to a givenpoint on the correlation surface is directly proportional to thedistance of that point from the stationary, centre motion vector. Themagnitude of the point on the correlation surface is multiplied by theweighting factor. For example, the gradient of the weighting functionmay be such that points plus or minus 32 pixels from the centre,stationary motion vector are multiplied hy a factor of three. In otherwords, as shown in FIG. 30, where the centre, stationary motion vectoris indicated by the black circle, the weighting function is an invertedcone which is centred on the centre, stationary motion vector.

After the correlatiou surface has been weighted, it is again passedthrough the threshold test and the rings test. If a minimum which passesboth these tests is determined, then it is assumed that this is a goodmotion vector, and it is flagged to indicate that it is a good motionvector, but that weighting was used. This flag is passed, together withthe motion vector to the motion vector reducer 45 (FIG. 4). If on theother hand, neither a good motion vector nor a best available motionvector can be determined, even after weighting, then a flag is set toindicate Lhat any motion vector passed to the motion vector reducer 45(FIG. 4) for this search block is a bad motion vector. It is necessaryto do this because bad motion vectors must not be used in the motionvector reduction process, but must be substituted as will be describedbelow.

Thus, in summary, the operation of the motion vector estimator 44 (FIG.4) is to derive from the correlation surface generated by the directblock marcher 43 (FIG. 4), the point of best correlation, that is theminimum. This minimum is then subjected to the threshold test and therings test, both of which the minimum must pass in order for it to beconsidered to represent the motion of the search block. It should,incidentally, be noted that the threshold used in the threshold test andthe rings test may be either absolute values or fractional values. Ifthe minimum fails either test, then the search block is grown, adetermined, and the threshold test and the new minimum is rings testre-applied. The most effective direction in which to grow the searchblock is determined from the shape of the correlation surface.

Referring initially to FIG. 4, the. process of motion vector reductionwill now be described. Using a HDVS, each search block is assumed to be32 pixels by 23 lines, which can be shown to lead to a possible maximumof 2451 motion vectors. The choice of the search block size is acompromise between maintaining resolution and avoiding an excessiveamount of hardware. If all these motion vectors were passed to themotion vector selector 46, the task of motion vector selection would notbe practicable, due to the amount of processing that would be required.To overcome this problem, the motion vector reducer 45 is providedbetween the motion vector estimator 44 and the motion vector selector46. The motion vector reducer 45 takes the motion vectors that have beengenerated by the motion vector estimator 44 and presents the motionvector selector 46 with only, for example, four motion vectors for eachsearch block in the frame, including those in border regions, ratherthan all the motion vectors derived for that frame. The effect of thisis two-fold. Firstly, this makes it much easier to choose the correctmotion vector, so long as it is within the group of four motion vectorspassed to the motion vector selector 46. Secondly, however, it alsomeans that if the correct motion vector is not passed as one of thefour, then the motion vector selector 46 is not able to select thecorrect one. It is therefore necessary to try to ensure that the motionvector reducer 45 includes the correct motion vector amongst thosepassed to the motion vector selector 46. It should also be mentionedthat although four motion vectors are passed by the motion vectorreducer 45 to the motion vector selector 46, only three of theseactually represent motion, the fourth motion vector always being thestationary motion vector which is included to ensure that the motionvector selector 46 is not forced into applying a motion vectorrepresenting motion to a stationary pixel. Other numbers of motionvectors can be passed to the motion vector selector 46, for example, inan alternative embodiment four motion vectors representing motion andthe stationary motion vector may be passed.

Hereinafter the term `sample block` refers to a block in a frame ofvideo in which each pixel is offered the same four motion vectors by themotion vector reducer 45. Thus, a sample block is the same a.s a searchblock before the search block has been grown. As shown in FIG. 31, in aframe of video the initial positions of the sample blocks and the searchblocks are the same.

The motion vector reducer 45 (FIG. 4) receives the motion vectors andthe flags from the motion vector estimator 44 (FIG. 4) and determinesthe qualitv of the motion vectors by examining the flags. If Lbe motionvector was not derived from an ambiguous surface, that is there is ahigh degree of confidence in it, then it is termed a good motion vector,but if a certain amount of ambiguity exists, then the motion vector istermed a bad motion vector. In the motion vector reduction process, allmotion vectors classed as bad motion vectors are ignored, because it isimportant that no incorrect motion vectors are ever passed to the motionvector selector 46 (FIG. 4), in case a bad motion vector is selectedthereby. Such selection would generally result in a spurious dot. in thefinal picture, which would be highly visible.

Each of the motion vectors supplied to the motion vector reducer 45(FIG. 4) was obtained from a particular search block, and hence aparticular sample block (FIG. 31), the position of these being notedtogether with the motion vector. Because any motion vectors which havebeen classed as bad motion vectors are ignored, not all sample blockswill have a motion vector derived from the search block at thatposition. The motion vectors which have been classed as good motionvectors, and which relate to a particular search block, and hence aparticular sample block, are called local motion vectors, because theyhave been derived in the area from which the sample block was obtained.In addition to this, another motion vector reduction process counts thefrequency at which each good motion vector occurs, with no account takenof the actual positions of the search blocks that were used to derivethem. These motion vectors are then ranked in order of decreasingfrequency, and are called common motion vectors. In the worst case onlythree common motion vectors are available and these are combined withthe stationary motion vector to make up the four motion vectors to bepassed to the motion vector selector 46 (FIG. 4). However, as there areoften more than three common motion vectors, the number has to bereduced to form a reduced set of common motion vectors referred to asglobal motion vectors.

A simple way of reducing the number of common motion vectors is to usethe three most frequent common motion vectors and disregard theremainder. However, the three most frequent common motion vectors areoften those three motion vectors which were initially within plus orminus one pixel motion of each other vertically and/or horizontally. Inother words, these common motion vectors were all tracking the samemotion with slight differences between them, and the other common motionvectors, which would have been disregarded, were actually trackingdifferent motions.

In order to select the common motion vectors which represent all or mostof the motion in a scene, it is necessary to avoid choosing globalmotion vectors which represent the same motion. Thus, the strategyactually adopted is first to take the three most frequently occurringcommon motion vectors and check to see if the least frequent among themis within plus or minus one pixel motion vertically and/or plus or minusone pixel motion horizontally of either of the other two common motionvectors. If it is, then it is rejected, and the next most frequentlyoccurring common motion vector is chosen to replace it. This process iscontinued for all of the most frequently occurring conunon motionvectors until there are either three common motion vectors which are notsimilar to each other, or until there are three or less common motionvectors left. However, if there are more than three common motionvectors left, then the process is repeated this time checking to see ifthe least frequent among them is within plus or minus two pixel motionvertically and/or plus or minus two pixel motion horizontally ofanother, and so on at increasing distances if necessary. These threecommon motion vectors are the required global motion vectors, and it isimportant to note that they are still ranked in order of frequency.

When considering the motion vector reduction process and the sampleblocks of a frame of video, it is necessary to look at three differenttypes of sample blocks. These types are related to their actual positionin a frame of video, and are shown in FIG. 32 as regions. Region Acomprises sample blocks which are totally surrounded by other sampleblocks and are not near the picture boundary. Region B contains sampleblocks which are partially surrounded by other sample blocks and are notnear the picture boundary. Finally, region C contains sample blockswhich are near the picture boundary. The motion vector reductionalgorittun to be used for each of these regions is different. Thesealgorithms will be described below, but firstly it should be reiteratedthat there exist good motion vectors for some of the sample blocks inthe frame of video, and additionally there are also three global motionvectors which should represent most of the predominant motion in thescene. A selection of these motion vectors is used to pass on threemotion vectors together with the stationary motion vector for eachsample block.

FIG. 33 illustrates diagrammatically motion vector reduction in theregion A. This is the most complex region to deal with, because it hasthe largest number of motion vectors to check. FIG. 33 shows a centralsample block which is hatched, surrounded by other sample blocks a to h.Firsfly, the locally derived motion vector is examined to see if it wasclassed as a good motion vector. If it was, and it is also not the sameas the stationary motion vector, then it is passed on. However, if itfails either of these tests, it is ignored. Then the motion vectorassociated with the sample block d is checked to see if it was classedas a good motion vector. If it was, and if it is neither the same as anymotion vector already selected, nor the same as the stationary motionvector, then it too is passed on. If it fails any of these tests then ittoo is ignored. This process then continues in a similar manner in theorder e, b, g, a, h, c and f. As soon as three motion vectors, notincluding the stationary motion vector, have been obtained, then thealgorithm stops, because that is all that is required for motion vectorselection for that sample block. It is, however, possible for all theabove checks to be carried out without three good motion vectors havingbeen obtained. If this is the case, then the remaining spaces are filledwith the global motion vectors, with priority being given to the morefrequent global motion vectors.

FIG. 34 illustrates motion vector reduction in the region B. Sampleblocks in the region B are the same as those in the region A, exceptthat they are not total ly surrounded by other sample blocks. Thus theprocess applied to these sample blocks is exactly the same as those forthe region A, except that it is not possible to search in all thesurrounding sample blocks. Thus as seen in FIG. 34, it is only possibleto check the motion vectors for the sample blocks a to e, and anyremaining spaces for motion vectors are filled, as before, with globalmotion vectors. Likewise, if the hatched sample block in FIG. 34 weredisplaced two positions to the left, then it will be seen that therewould only be three adjacent surrounding blocks to be checked beforeresorting to global motion vectors.

FIG. 35 illustrates motion vector reduction in the region C. This is themost severe case, because the sample blocks neither have a locallyderived motion vector nor do they have many surrounding sample blockswhose motion vectors could be used. The simplest way of dealing withthis problem is simply to give the sample blocks in the region C theglobal motion vectors together with the stationary motion vector.However, this is found to produce a block-like effect in the resultingpicture, due to the sudden change in the motion vectors presented forthe sample blocks in the region C compared with adjoining sample blocksin the region B. Therefore a preferred strategy is to use for the sampleblocks in the region C the sample motion vectors as those used forsample blocks in the region B, as this prevents sudden changes.Preferably, each sample block in the region C is assigned the samemotion vectors as that sample block in the region B which is physicallynearest to it. Thus, in the example of FIG. 35, each of the hatchedsample blocks in the region C would be assigned the same motion vectorsas the sample block a in the region B, and this has been found to giveexcellent results.

Referring again to FIG. 4, the purpose of the motion vector selector 46is to assign one of the four motion vectors supplied thereto to eachindividual pixel within the sample block. In this way the motion vectorscan be correctly mapped to the outline of objects. The way in which thisassignment is effected is particularly intended to avoid the possibilityof the background surrounding fine detail from producing a better matchthan that produced by the correct motion vector. To achieve this themotion vector selection process is split into two main stages. In thefirst stage, motion vectors are produced for each pixel in the inputframes. In other words, there is no attempt to determine the motionvector values for pixels at the output frame positions. The second stageuses the motion vector values produced by the first stage to determinethe motion vector value for each pixel in the output frame.

Referring now to FIG. 36, each pixel of the input frame 2 is tested forthe best luminance value match with the previous and following inputframes 1 and 3 of video data, using each of the four motion vectorssupplied. The pixel luminanee difference is determined as: ##EQU5##where: P1_(nm) is the luminance value of a frame 1 pixel within a 4×4block of pixels surrounding the pixel whose location is obtained bysubtracting the coordinates of the motion vector being tested from thelocation of the pixel being tested in frame 2

P2_(nm) is the luminanee value of a frame 2 pixel within a 4×4 block ofpixels surrounding the pixel being tested

P3_(nm) is the luminance value of a frame 3 pixel within a 4×4 block ofpixels surrounding the pixel whose location is obtained by adding thecoordinates of the motion vector being tested to the location of thepixel being tested in frame 2

The minimum pixel difference then indicates the best luminance match andtherefore the correct motion vector applicable to the pixel beingtested. if the correct motion vector is not available, or titere areuncovered or covered areas, referred to in more detail below, then agood match may not occur.

The indication of a poor match is achieved when the average pixeldifference within the block of pixels being used is above a certainthreshold. This threshold is important, because high frequency detailmay produce a poor match even when the correct motion vector is tested.The reason for this poor match is the possibility of a half pixel errorin the motion vector estimate. To determine what threshold shouldindicate a poor match, it is necessary to relate the threshold to thefrequency content of the picture within the block of data whichsurrounds the pixel for which the motion vector is required. To achievethis, an auto-threshold value is determined where the threshold valueequals half the maximum horizontal or vertical pixel luminancedifference abouL the pixel being tested. To ensure that the thresholdvalue obtained is representative of the whole block of data which iscompared, an average value is obtained for the four central pixels a 4×4block used.

Referring to FIG. 38, which shows a 4×4 block, the required thresholdvalue T is given by:

    T=(T1+T2+T3+T4)/8

where T3, for example, is determined as indicated in FIG. 39 as equal tothe maximum of the four pixel luminance difference values comprising:

the two vertical differences |B2-B3| and |B4-B3|, and

the two horizontal differences |A3|B3| and |C3-B3|

In this way a frame of motion vectors is obtained for input frame 2, andin a similar manner a frame of motion vectors is obtained for inputframe 3 as indicated in FIG. 37.

Apart from scene changes, it is the phenomenon of uncovered/coveredsurfaces that causes a mis-match to occur in above first stage of motionvector selection. If an object, say a car, drives into a tunnel, thenthe car has become covered, while when it drives out, the car isuncovered. If the part of the car that was recovered in frames 1 and 2is covered in frames 3 and 4, then the basic vector selection process isnot able to determine the correct vector. Moreover, whilst the car goinginto the tunnel becomes covered, the road and objects behind the car arebeing uncovered. Likewise the car leaving the tunnel is being uncovered,but the road and obiects behind the car are being covered. In generaltherefore both covered and uncovered objects will exist at the sametime. The end of a scene will also have a discontinuation of motion thatis similar to an object becoming covered. In an attempt to determine amotion vector even in such circumstances, the luminance value blockmatch is reduccel to a two frame match, instead of the three frame matchof FIGS. 36 and 37. The frame that the motion vectors are required for(say frame 2) is block-matched individually to the previous and the nextframe (frame 1 and frame 3 respectively, in the case of frame 2), usingthe four motion vectors supplied. The motion vector which produces thebest match is chosen as the motion vector applicable to time pixel beingtested. in this case, however, a flag is set to indicate that only a twoframe match was used.

Particuarly with integrating type television cameras, there will besituations where no match occurs. If an object moves over a detailedbackground, then an integrating camera will produce unique portions ofpicture where the leading and trailing edges of the object are mixedwith the detail of the background. In such circumstances, even the twoframe match could produce an average pixel difference above thethreshold value. In these cases the motion vector value is set to zero,and an error flag is also set.

The second stage of motion vector selection makes use of the two framesof motion vectors, derived by the first stage. One frame of motionvectors (input frame 2) is considered to be the reference frame, and thefollowing frame to this (input frame 3) is also used. The output frameposition then exists somewhere between these two frames of motionvectors. Referring to FIG. 40, for each output pixel position the fourpossible motion vectors associated with the sample block of input frame2, are tested. A line drawn through the output pixel position at theangle of the motion vector being tested will point to a position on boththe input frame 2 and the input frame 3. In the case of odd value motionvectors, for example, 1, 3 and 5, a point midway between two input framepixels would be indicated in the case where the output frame isprecisely half way between the input frames 1 and 2. To allow for thisinaccuracy, and also to reduce the sensitivity to individual pixels, a3×3 block of motion vectors is acquired for each frame, centred on theclosest pixel position. In effect a block-match is then performedbetween each of the two 3×3 blocks of motion vectors and a blockcontaining the motion vector being tested. The motion vector differenceused represents the spatial difference of the two motion vector valuesas given by:

    √((x1-x2).sup.2 +(y1-y2).sup.2)

where:

x1 and y1 are the Cartesian coordinates of the motion vector in one ofthe blocks

x2 and y2 are the Cartesian coordinates of the motion vector beingtested

An average vector difference per pixel is produced as a result of theblock match.

A motion vecLor match is first produced as above using only motionvector values which were calculated using three input frames; that is,input frames 1, 2 and 3 for input frame 2 (FIG. 36), and input. frames2, 3 and 4 for input frame 3 (FIG. 37), and the result is scaledaccordingly. Preferably there are at least four usable motion vectors inthe block of nine. When both the motion vector block of frame 2 andframe 3 can be used, the motion vector difference values are made up ofhalf the motion vector difference value from frame 2 plus ball themotion vecLor difference value from frame 3. Whichever motion vectorproduces the minimum motion vector difference value using the abovetechnique is considered to be the motion vector applicable to the outputpixel being tested. If the motion vector difference value produced bythe three frame match input motion vector (FIGS. 36 and 37 is greaterthan unity, then a covered or uncovered surface has been detected, andthe same process is repeated, but this the ignoring the error flags.That is, the motion vector values which were calculated using two inputframes are used. Theoretically this is only necessary foruncovered/covered surfaces, although in fact improvements can beobtained to the picture in more general areas.

If after both of the above tests have been performed, the minimum motionvector match is greater than two, the motion vector value is set tozero, and an error flag is set for use by the motion vector postprocessor 47 (FIG. 4).

Following motion vector selection, there will almost certainly be in anyreal picture situation, some remaining spurious motion vectorsassociated with certain pixels. FIGS. 41 to 46 show what are taken to bespurious motion vectors, and in each of these figures the trianglesrepresent pixels having associated therewith the same motion vectors,whilst the stars represent pixels having associated therewith motionvectors different those associated with the surrounding pixels, and thecircle indicates the motion vector under test.

FIG. 41 shows a point singularity where a single pixel has a motionvector different from those of all the surrounding pixels.

FIG. 42 shows a horizontal motion vector impulse, where threehorizontally aligned pixels have a motion vector different from those ofthe surrounding pixels.

FIG. 43 shows a vertical motion vector impulse where three verticallyaligned pixels have a motion vector different from those of thesurrounding pixels.

FIG. 44 shows a diagonal motion vector impulse where three diagonallyaligned pixels have a motion vector different from those of all thesurrounding pixels.

FIG. 45 shows a horizontal plus vertical motion vector impulse, wherefive pixels disposed in an upright cross have a motion vector differentfrom those of all the surrounding pixels.

FIG. 46 shows a two-diagonal motion vector impulse where five pixelsarranged in a diagonal cross have a motion vector different from thoseof all the surrounding pixels.

It is assumed float pixel motion vectors which fall into any of theabove six categories do not actually belong to a real picture, and are adirect result in of an incorrect motion vector selection. If such motionvectors were used cturing the interpolation process, then they would beIikely to cause dots on the final output picture, and it is thereforepreferable that such motion vectors be identified and eliminated. Thisis done using an algorithm which will detect and flag all of the abovemotion vector groupings.

The algorithm uses a two-pass process, with each pass being identical.The need for two passes will become apparent. FIG. 47, to whichreference is made, shows an array of pixels, all those marked with atriangle having the same motion vector* associated therewith. The blockof nine pixels in the centre has motion vectors designated vector 1 tovector 9 associated therewith, which motion vectors may or may not bethe same. Vector 5 is the motion vector under test.

In the first pass, vector 5 is checked to determine whether it is thesame as, or within a predetermined tolerance of:

firstly

vector 1 or vector 3 or vector 7 vector 9

and secondly

vector 2 or vector 4 or vector 6 or vector 8

This checks to see if vector 5 is the same as at least one of itshorizontal or vertical neighbouts, and tire same as at least one of itsdiagonal neighbouts. If this is not the case, then a flag to set toindicate that pixel 5 is bad.

The first pass will flag as bad those motion vectors relating to pointsingularities, horizontal motion vector impulses, vertical motion vectorimpulses, diagonal motion vector impulses and two diagonal motion vectorimpulses (FIGS. 41 to 44 and 46), but not the motion vectorscorresponding to horizontal plus vertical motion vector impulses (FIG.45) for which pass 2 is required. The second pass checks for exactly thesame conditions as in the first pass, but in this case motion vectorswhich have already been flagged as bad are not included in thecalculation. Thus, referring to FIG. 45, after the first pass only thecentre motion vector is flagged as bad, but after the second pass allfive of the motion vectors disposed in the upright cross are flagged asbad.

Having identified the bad motion vectors, it is then necessary to repairthem, this also being effected by the motion vector post processor 47(FIG. 4). Although various methods such as interpolation or majorityreplacement can be used, it is has been found that in practice simplereplacement gives good results. This is effected as follows (and itshould be noted that the `equals` signs mean not only exactly equal to,but also being within a predetermined tolerance of):

If vector 5 is flagged as bad then it is replaced with:

vector 4 if (vector 4 equals vector 6)

else which vector 2 if (vector 2 equals vector 8)

else with vector 1 if (vector 1 equals vector 9)

else with vector 3 if (vector 3 equals vector 7)

else do nothing

Referring again to FIG. 4, the finally selected motion vector for eachpixel is supplied by the motion vector post processor 47 to theinterpolator 48, together with the progressive scan converted frames at60 frames per second from the progressive scan converter 42. Theinterpolator 48 is of relatively simple form using only two progressivescan converted frames, as indicated in FIG. 48. Using the temporalposition of the output frame relative to successive input frames, frame1 and frame 2, and the motion vector for the pixel in the output frame,the interpolator 48 determines in known manner which part of the firstframe should be combined with which part of the second frame and witIxwhat weighting to produce the correct output pixel value. In otherwords, the interpolator 48 actaptively interpolates along the directionof movement in dependence on the motion vectors to produce motioncompensated progressive scan frames corresponding to 24 frames persecond. Although the motion vectors have been derived using onlyluminance values of the pixels, the same motion vectors are used forderiving the required output pixel chrominance values. An 8×8 array ofpixels are used from each frame to produce the required output. Thus theinterpolator 48 is a two-dimensional, vertical/horizontal, interpolatorand the coefficients used for the interpolator 48 may be derived usingthe Remez exchange algorithm which can be found fully explained in`Theory and application of digital signal processing`, Lawrence RRabiner, Bernard Gold. Prentice-Hall Inc., pages 136 to 140 and 227.

FIG. 48 shows diagrammaticaIly the interpolation performed by theinterpolator 48 (FIG. 4) for three different cases. The first case,shown on the left, is where there are no uncovered or covered surfaces,the second case, shown in the centre, is where there is a coveredsurface, and the third case, shown on the right, is where there is anuncovered surface. in the case of a covered surface, the interpolationuses only frame 1, whilst in the case of an uncovered surface, theinterpolation uses only frame 2.

Provision can be made in the interpolator 48 to default to non-motioncompensated interpolation, in which case the temporally nearestprogressive scan converted frame is used.

In the arrangement described above, particularly with reference to FIG.2, a 60 field/s, 30 frame/s, 2:1 interlaced format signal is convertedto 24 frame/s 1:1 progressive format by:

a) supplying the interlaced signal to the standards converter atone-eighth speed with each input field repeated eight times;

b) developing, for each input field, a progressive format frame in theprogressive scan converter 42;

c) for each ten repeated input frames, developing an output frame byappropriate interpolation in the interpolator 48 between the twocurrently developed progressive formal frames in dependence upon thesupplied motion vector, with the deveIoped output frame being repeatedten times; and

d) recording one in every ten output frames.

It will therefore be appreciated that the standards converter operatesat one-eighth of real-time, and that, for every five input interlacedframes, four output frames will be produced, thus giving the 30 frame/sto 24 frame/s conversion.

In some applications, for example where further processing is to becarried out on the 24 frame/s 1:1 format signal, or where it is desiredto record the 24 frame/s format signal on standard HDVS recordingequipment and replay it, it is beneficial to use a modified 24 frame/sformat employing a "3232 pulldown" sequence. For further description ofthe 3232 pulldown sequence, reference is directed to patent applicationGB 9018805.3, the content of which is incorporated herein by reference.

The correlation between a series of four frames of a 24 frame/s 1:1format signal and a 30 frame/s, 60 field/s, 3232 pulldown format signalis shown in FIG. 49. The first frame 1 is used to produce the firstthree fields, with odd field 3 being a phantom field. Frame 2 proctucesthe next two fields 4 and 5. Frame 3 produces the next three field 6 to8, with even field 8 being a phantom field, and the last frame 4 in thesequence produces the last two field is 9 and 10.

It is desirable to modify the system described with reference to FIGS. 1to 48 so that it can convert a 60 field/s, 30 frame/s 2:1 interlacedvideo signal to a 60 field/s 3232 pulldown sequence with motioncompensation, and this can be achieved in a remarkably simple way byoperating the frame recorder 13 and VTR 14 of FIG. 2 at one-eighthspeed, rather than one-tenth speed (while still maintaining theten-frame repeat of the output from the standards converter 12) with theperiod between recorded fields being 9, 7, 9, 7 . . . field periods.This scheme is illustrated in FIG. 50 for five input frames.

As shown in columns A and B of FIG. 50, the two fields O/E of each ofsix input frames 1 to 6 are repeated eight times. Certain of these inputfields then used by the progressive scan converter 42 to producerespective progressive scan frames as shown in column D, the progressivescan frames being stored alternately in a pair of frame stores. Certainof the stored progressive scan frames are then used alone or in pairs bythe interpolator 48 to produce four fields, each of which is output bythe interpolator 48 ten times as shown in column E. A field is thenrecorded by the frame recorder 13 from every fourth frame of column E,the recorded fields being alternately odd and even with the periodbetween recorded fields being alternately 9/60s and 7/60s, as shown incolumn F, and these fields are then recorded on the VTR 14. Thus, whenthe recording is played back at normal speed, the sequence shown in thecolumn G of FIG. 50 is produced, with the ten recorded fields in 3232pulldown format 1 odd, 1 even, 2 odd, 2 even . . . 5 even being derivedfrom frame A, repeats 1, 5 and 9; frame 3, repeats 3 and 7; frame C,repeats 1, 5 and 9; and frame D, repeats 3 and 7 of the 24 Hz 1:1 formatsequence.

As an alternative to the increased recording speed (one-eighth, ratherthan one-tenth) modification for producing the 3232 pull down sequence,the progressive scan frames may be written into the frame recorder 13 atone-tenth speed and then read out in 3232 sequence for supply to the VTR14. This modification is illustrated in FIG. 51. Column E corresponds tocolumn E described above with reference FIG. 50. One repeat of eachprogressive scan frame is written to the recorder 13, as shown in columnF. Odd and even fields of these frames are then read out from therecorder 13 and recorded by the VTR 14 in the following sequence 1 odd,1 even, 1 odd, 2 even, 2 odd, 3 even, 3 odd, 3 even, 4 odd, 4 even, asshown in column G, to produce fields odd, 2 even, 3 odd, 4 even . . . 9odd, 10 even of the 3232 pulldown sequence, where 3 and 8 are thephantom fielets. This modification has an advantage over that describedwith reference to FIG. 50 in that less storage space is used in therecorder 13, and also conversions will require fewer stops and starts ofthe VTR 14.

It is desirable also to he able to use the system described above toconvert 30 Hz 1:1 format fiim material into 60 field/s 2:1 interlacedHDVS format using motion compensated interpolation. This could beachieved, as shown in FIG. 52, by producing the odd fields of convertedsignal directly from the input frames, anct by producing the even fieldsof the converted signal by motion compensated interpolation betweensuccessive fields with equal temporal offsets. However, it is consideredthat such a scheme would cause uneven spatial response of the fieldpairs and noise level modulation in the case of noisy source materialbecause the directly produced output fields would contain the sourcenoise, but the interpolated output fields would have reduced noise dueto the interpolator action. In order to avoid these problems, a temporalinterpolation scheme, as shown in FIG. 53A, is adopted, in which the oddfields are temporally interpolated one-quarter of the way between apreceding and a succeeding frame of the source material and the evenfields are temporally interpolated three-quarters of the way betweenthose two frames.

Thus, referring to FIG. 53B, if a pixel in an odd output field AObetween input frames 1,2 has a position (x,y) and a motion vector (m,n),the value of that pixel is obtained by averaging the value of the pixel(or patch) in input: frame 1 at location (x-(m/4), y-(n/4)) with thevalue of the pixel (or patch) in input frame 2 at location (x+(3m/4),y+(3n/4)). On the other hand, for the corresponding even output fieldAt, the output pixel value is obtained by averaging the value of thepixel (or patch) in input frame 1 at (x-(3m/4), y-(3n/4)) with the valueof the pixel (or patch) in input frame 2 at (x-(m/4), y+(n/4)), as shownin FIG. 53C.

In order to achieve this, the arrangement described with reference toFIGS. 1 to 48 is modified or used in the manner which will now bedescribed with reference to FIG. 54.

The frames of the 30 Hz 1:1 format film, three of which are shown incolumn A in FIG. 54, are captured by an HD film scanner and converted tovideo format on tape, as shown in column B. Altbough odd "0" and even"E" fields are shown for each frame 1, 2, 3 in column B, it should beremembered that the image data in the two fields of each frame are nottemporally offset in tire source image. The tape is then played back atone-twentieth speed by the VTR 11 (FIG. 2) with the two fields of eachframe repeated twenty times, as shown by column C. Because the twofields of each frame are derived from a progressive format source, thereis no temporal offset between them, and therefore the progressive scanconverter 42 (FIG. 4) is bypassed or operated in a previous fieldreplacement mode, in order to reconstruct the original frames, so thattwo consecutive frames are available at a time, as shown in column D.The frames which are input to the direct block marcher 43 andinterpolator 48 are thus a direct combination of the respective twofields. In order to produce a pixel in an odd output field, theinterpolator temporally interpolates in the ratio 3/4:1/4 between theprevious frame (e.g. 1) and tire current frame (e.g. 2). However, forpixels in an even field, the interpolation is in the ratio 1/4:3/4between the previous frame 1 and the current frame 2, and theinterpolated frames are each repeated twenty times as shown in column E.Every twentieth frame is written to the frame recorder 13 (FIG. 2) asrepresented in column F of FIG. 54 and is then recorded by the VTR 14,so that when the recording is played back a 60 field/s 2:1 format signalis produced at normal speed, as represented by column G in FIG. 54.

It is also desirable to be able to use the system described above toconvert 24 Hz 1:1 format film material to 60 field/s 2:1 interlaced HDVSformat. In order to do this, firstly the 24 Hz 1:1 format frames arecaptured by an HD film scanner and converted to video format on tape.The temporal interpolation scheme which is then used is shown in FIG.55A, It will be noted that the interpolation sequence repeats afterevery 4 frames (1 to 4) of the 24 Hz 1:1 format signal and after every 5frames, or 10 fields (A/odd to E/even), of the 60 field/s 2:1 interlacedHDVS signal.

If a pixel in all output field has a moLion vector (m,n), then theoffsets between the location of that pixel in the output field and thelocations in the respective two input frames of the pixels (or patches)used to derive the value of the output pixel are as follows for each ofthe ten output frames in the series:

    ______________________________________                                        Output      First Input Frame                                                                           Second Input Frame                                  Frame/Field and Offset    and Offset                                          ______________________________________                                        A/odd       1 (0, 0)                                                          A/even      1 (-0.4m, -0.4n)                                                                            2 (0.6m, ).6n)                                      B/odd       1 (-0.8m, -0.8n)                                                                            2 (0.2m, 0.2n)                                      B/even      2 (-0.2m, -0.2n)                                                                            3 (0.8m, 0.8n)                                      C/odd       2 (-0.6m, -0.6n)                                                                            3 (0.4m, 0.4n)                                      C/even      3 (0, 0)                                                          D/odd       3 (-0.4m, -0.4n)                                                                            4 (0.6m, 0.6n)                                      D/even      3 (-0.8m, -0.8n)                                                                            4 (0.2m, 0.2n)                                      E/even      4 (-0.2m, -0.2n)                                                                            5 (0.8m, 0.8n)                                      E/even      4 (-0.6m, -0.6n)                                                                            5 (0.4m, 0.4n)                                      ______________________________________                                    

In order to achieve such an interpolation sequence, the system describedabove with reference to FIGS. 53 and 54 is modified as follows,referring to FIG. 56. In FIG. 56, for clarity, the separate fields offrames are not shown. The 24 Hz 1:1 format frames (five of which areshown in column A of FIG. 56), after capturing by the HD scanner (columnB) are reproduced at one-twentyfifth speed by the VTR 11 (FIG. 2) withthe two fields of each frame being repeated twenty-five times, as shownin colunn C. As described with reference to FIG. 54, the progressivescan converter 42 is operated in a previous field replacement mode toreconstruct the original frames, and so that three frames are availableat a time, as shown in column D. The interpolator 48 then performs thenecessary interpolation to produce the frames A to E, as shown in columnE. It will be noted from the table above, that input frames 1 and 2 arerequired to produce the two fields of output frame A; input frames 1, 2and 3 are required for the fields of output frame B; input frames 2 and3 are required for the fields of output frame C; input frames 3 and 4are required for the fields of output frame D; and input frames 4 and 5are required for the fields of output frame E. The frames thus formed bythe interpolator are repeated twenty times, as shown in column E. One inevery twenty interpolated frames is written to the frame recorder 13, asrepresented by column F, and recorded in tape by the VTR 14 so that whenthe tape is played back at normal speed, a 60 field/s 2:1 interlacedHDVS is produced at normal speed, as represented by column G.

The system should also desirably be able to handle 24 Hz 1:1 materialprovided in 3232 pulldown format. in order to do this, the arrangementof FIG. 56 is utilised except that the input tape from the VTR 11 is runat one-twentieth speed, rather than 1/25 speed, and the inputframes/fields are repeated twenty times, rather than twenty-five timesbefore the next field is input. The frames of column D are stillproduced one for every 50 repeated input fields, and therefore thephantom fields of the 3232 pulldown format can be ignored. Accordingly,the system operates somewhat similarly to that described with referenceto FIG. 54 relating to conversion of 30 Hz 1:1 format film material to a60 field/s 2:1 interlace format HDVS.

A modification may be made to the arrangement described with referenceto FIGS. 55A and 56 so that it can convert to 30 Hz 1:1 format, ratherthan 60 field/s 2:1 interface format. With this modification, the outputframes are derived from the input frames as shown in FIG. 55B. In orderto achieve this modification, the only change necessary to make thescheme: of FIG. 56 is to modify the action of the interpolator so that,for a pixel in an output frame having a motion vector (m,n), the offsetsbetween the location of that pixel in the output frame and the locationsin the respective two input frames of the pixels (or patches) used totierre the value of the output pixel are as follows for each of the fiveoutput frames in the series:

    ______________________________________                                        Output      First Input Frame                                                                           Second Input Frame                                  Frame       and Offset    and Offset                                          ______________________________________                                        A           1 (0, 0)                                                          B           1 (-0.8m, -0.8n)                                                                            2 (0.2m, 0.2n)                                      C           2 (-0.6m, -0.6n)                                                                            3 (0.4m, 0.4n)                                      D           3 (-0.4m, -0.4n)                                                                            4 (0.6m, 0.6n)                                      E           4 (-0.2m, -0.2n)                                                                            5 (0.8m, 0.8n)                                      ______________________________________                                    

It, is also desirable that the system described above should be able toconvert a 60 field/s 2:1 interlace HDVS format signal to a 30 Hz 1:1progressive formal for use by the EBR 35 (FIG. 3) in producing 30 Hzfilm from the video signal. Such a 30 Hz signal could be produced bycompositing each frame from two adjacent fields in the HDVS, but thiswould have the effect of producing double-imaging in the 30 Hz signal,due to the temporal offset between the two fields making up each frame.

The progressive scan arrangement described above with reference to FIGS.4 to 14 can be employed to good effect, to blend different proportionsof interframe and intrafield interpolated images depending on the amountof motion locally in the image, and thus provide a motion adapted 30 Hzsignal. However, when the source image is noisy or there is an incorrectassessment. by the motion adaptive process, the output image will tendto be intrafield interpolated more than is necessary, and thus will losevertcal detail and have more alias components present. In thesecircumstances, it is possible that the output image quality will beimproved by additionally using the motion compensation techniquedescribed above with reference to FIGS. 15 to 48. This will allow twoadjacent 30 Hz 1:1 frames to be combined via temporal interpolation andprovide cancellation of vertical alias components in static orhorizontally moving parts of the image.

The technique using the motion adaptive technique only, is shown inFIGS. 57 and 58. Referring firstly to FIG. 57, and as described above,each frame A, B, C in the output format is produced by differentproportions of three fields (e.g. 1'0', 1'E', 2'0'; 2'0', 2'E', 3'0';3'0', 3'E', 4'0') in the input format. Referring now to FIG. 58, the 60field/s 2:1 HDVS format signal (column A) is reproduced at one-eighthnormal speed by the VTR 11 (FIG. 2), and each frame/field is repeatedeight times, as shown in column B. The progressive scan converter 42(FIG. 4) loads fields into its frame stores so that three consecutivefields are available at a the, as shown in column C. The progressiveformat frames are formed from the triplets of input fields, as shown incolumn D, and these progressive format frames are each repeated eightthes. Every eighth frame (column E) is recorded by the frame recorder 13and then by the VTR 14 (FIG. 2) and thus the recorded signal, whenreproctuced at normal speed is in 30 Hz 1:1 format (column F). In thismotion adaptive mode, the progressive scan converter 42 (FIG. 4) isemployed, but the motion compensation components 43 to 48 are not.Therefore, the interpolator 48 is set to provide no temporal offsetbetween its input and output frames.

When motion compensation is selected, the operation is as shown in FIGS.59 and 60. The upper part of FIG. 59 is similar to FIG. 57, except thattwice as many progressive format frames are produced. The frames soformed then undergo the motion compensation operation to form the outputframes which are temporally offset by half a frame period from franmsbefore motion compensation.

Referring in particular to FIG. 60, the 60 field/s 2:1 HDVS formalsignal (column A) is reproduced at one-tenth normal speed hy the VTR 11(FIG. 2), and each frame/field is repeated ten times, as shown in columnB. As before, the progressive scan converter 42 loads fields into itsframes stores so that three consecutive input fields are available at atime, as shown in column C. The progressive scan frames 1", 2', 2", 3' .. . are then formed from the triplets of input fiel. ds, and are loadedinto frame stores so that the interpolator 48 has available twoconsecutive progressive format frames at a time, as shown in column D.Temporally adjacent pairs of these progressive format frames are thenused by the interpolator 48 to produce a frame which is interpolatedwith equal temporal offset between the two input frames. The framesproduced by the interpolator 48 are repeated ten times as shown incolumn E, and every tenth frame (column F) is recorded by the framerecorder 13 and VTR 14 (FIG. 2) , so that the thus recorded signal, whenreproduced at normal speed, is in 30 Hz 1:1 format (column G).

In the arrangement described above with reference to FIGS. 1 to 48 forconverting 60 field/s 2:1 interlace HDVS to 24 Hz 1:1 film formal, theouLput frames are either temporally aligned with respective input framesor temporally offset by one half of an input field period (1/120s). Inthe case of temporal alignment, the output frame is based upon therespective progressive format frame output from the progressive scanconverter 42 (FIG. 4), whereas in the case of a temporal offset, eachpixel in the output frame is 1/2:1/2 interpolated between pixels orpixel patches in prececiing and succeeding progressive format framesoutput from the progressive scan converter, with spatial offset betweenthe pixels or patches in the source frames and the pixel in the outputframe being dependent upon the motion vector suppllied by the processor47 (FIG. 4).

FIG. 61A shows the case where there is no temporal offset and a pixel atlocation (x,y) in the outpul frame has a motion vector (m,n). This pixelis derived from the pixel at location (x,y) or a patch centred on (x,y)in input frame 1 to the interpolator, and the motion vecLor and thecontent of input frame 2 are not employed.

FIG. 61B shows the case where there is a half field period temporaloffset and a motion vector of (m,n)=(2,2) for an output pixel atlocation (x,y). The value of this pixel is derived by equalinterpolation between the pixel at (or patch centred on) location (x,y)-1/2(m,n)=(x-1,y-1) in input frame 1 and the pixel/patch at location(x,y)+1/2(m,n)=(x+1,y+1) in input frame 2.

It will be appreciated that the components of the motion vector (m,n)are integers and need not be even integers. FIG. 61C shows the case of amotion vector (2,1). The required pixels or patches in the input frames1 and 2 are at locations (x-q ,y-1/2and (x+1,y+2) , respectively, whichare half-way between actual pixel positions in the input frames.

In order to acqui re the required pixel values from the input frames, an8×8 patch (as described above with reference to FIG. 48) or a 7×7 patchas shown in FIGS. 62A to D is used around the required pixel location,and there is an offset of (0,0) (FIG. 62A), (1/2,0) (FIG. 62B), (0,1/2)(FIG. 62C), or (1/2,1/2) (FIG. 62D) between the centre pixel of thepatch and the pixel location (marked "o" in FIGS. 62A to D) determinedby the interpolator. To determine the value of the pixel "o", spatialinterpolation coefficients are applied to the 49 pixels in the patch,the sets of coefficients being slightly different for the four casesshown in FIGS. 62A to D, although the coefficients for the cases ofoffset (1/2,0) and (0,1/2) may be symmetrical about the x=y diagonaI ofthe patch.

A problem which can arise with the arrangement described above is thatthe magnitude responses for the four different spatial interpolationcases of FIGS. 62A to D can be different and produce modulation of thepicture detail as the different responses are cycled. In order to avoidthis problem, the arrangement now described with reference to FIGS. 63and 64 may be employed. The locations of the required pixels/patches inframes 1 and 2 are derived in a similar manner to that described withreference to FIG. 61 except that an addition offsel. of (-1/4,-1/4) isincluded. Thus, as shown in FIG. 63A, for example, for a temporallyaligned output frame, a pixel at locations (x,y) in the output frame isbased on an interpolated value of a pixel at location (x-1/4,y-1/4) ininput frame 1, rather than location (x,y). When taking into accounttemporally aligned and temporally offset output frames, and even and oddmotion vectors, a required pixel location in an input frame can alwaysbe considered to have an offsel of (-1/4,-1/4), (1/4,-1/4), (-1/4,1/4)or (1/4,1/4) from the centre pixel of a 7×7 patch from which the valueof the required pixel can be calculated by spatial interpolation, asshown in FIGS. 64A to 64D. It should be noted that these offsets arerotationally symmetrical about the centre pixel of the patch, andaccordingly the four sets of spatial interpolation coefficients for thecases shown in FIGS. 64A to D can also be chosen to be rotationallysymmetrical, thus avoiding different magnitude responses and picturedetail modulation. It will be appreciated that the above arrangementproduces global offset in the output frame of (-1/4,-1/4) pixels, butthis is negligible.

From the above it will be appreciated that, with conversion from 60field/s 2:1 interlace HDVS to 24 Hz 1:1 film format, every other outputframe from the interpolator 48 is produced from one of the progressiveframes input to the interpolator, and the alternate output frames areproduced by motion compensation between two progressive frames input. tothe interpolator. This can result in (a) perspective changes not beingsatisfactorily merged, (b) alias effects when the adaptive progressivescan conversion fails due to noise, and (c) noise level modulation whenthe input image is noisy. As regards point (b), when adaptiveprogressive scan fails due to noise, the progressive scan frames areproduced by intrafield interpolation. Of such a frame is directly outputby the interpolator 48, stationary images would appear heavily aliased.

In order to reduce these problems, in the case of a temporally alignedoutput frame from the interpolator 48, the frame is produced by equalsumming of the two respective input frames to the interpolator, but withthe motion vector being used to determine the spatial offset between therespective pixel in the output frame and the pixel/patch to be used inframe 2, without there being army spatial offset dependent upon themotion vector belween the pixel in the output frame and the pixel/patchto be used in frame 1. This scheme is illustrated in FIG. 65, incombination with the quarter pixel offset. scheme described above withreference to FIGS. 63 and 64. In the example given, a pixel at location(x,y) in the output frame has a motion vector (2,1). Accordingly, thenotional source pixel to be used from frame 1 has a location (x-1/4,y-1/4), and therefore the 7×7 patch centred on location (x,y) is usedwith the set of spatial interpolation coefficients for an offset of(-1/4,-1/4) (FIG. 64A). On the other hand, the notional source pixel tobe used from frame 2 has a location (x+7/4, y+3/4), and therefore the7×7 patch to be used is centred on location (x+2, y+1) with the spatialinterpolation coefficient set also for an offset (-1/4,-1/4).

By producing each pixel in the output frame by equal suntuning from twoinput frames, whether or not there is a temporal offset between theoutput frame and the input frames, alias is removed, because the frame 2alias will always be in antiphase to the frame 1 alias as long as theinterfield motion is an exact multiple of lines. As synthesised linesare mixed with non-synthesised lines in this scheme, an improvedvertical response is also produced. A further advantage is that if thereis noise in the input image, the noise will not be modulated, unlike thecase where every other output frame is derived from only one of theinput frames.

There now follows a description of a particular system using theapparatus described above which is used primarily for transfer from 24Hz the film to 24 Hz film permitting HDVS post production. Due to thecomplexity of motion compensated interpolation processing, the equipmenttherefor tends to be large and expensive. In addition, there is always arisk that processing artifacts may be introduced into the video signal.For these reasons, it is desirable that only a single stage of motioncompensated processing should be used between the source(s) and primarydistibution path, if at all possible.

Referring to FIG. 66, the first source is 24 Hz film material 100, andthe primary distribution medium is also 24 Hz film material 102. Thesource film 400 is read by a high definition film scanner 104, whichincorporates a 3232 pulldown system 106 to provide a high definition 60Hz 3232 format signal which is recorded by VTR 108.

The second source is a camera 140 which provides a 60 field/s 2:1interlace HDVS signal to a VTR 112. The VTR 112 can reproduce the HDVSsignal for input to a standards converter 414 which converts the 60field/s 2:1 HDVS signal to a 60 Hz 3232 format signal which is recordedon VTR 116. The standards converter is therefore as described above withreference to FIGS. 49 to 51. Upon reproduction of the 60 field/s 3232format signals by the VTRs 108, 116, these signals can be integrated inthe HDVS post production system 118 to produce an output signal in 60 Hz3232 format which is recorded by VTR 120. For further informationconcerning post-production with a 3232 pulldown format signal, referenceis directed to patent application GB9018805.3, the content of which isincorporated herein by reference. Upon reproduction of the signal by theVTR 120, the primary path is via a drop field device 122 which convertsthe signal to 24 Hz 1:1 format, which is then used by the electron beamrecorder (EBR) 124 to generate the film 102. Secondary distributionpatIls are provided by a VTR 126 which can record the output signal fromthe drop field device 122, and the signal on reproduction is then fed toa 625 lines converter 128 which produces a 50 field/s 2:1 interlace 625line signal and/or to a high definition 50 Hz 2:1/1:1 converter 130which produces a 50 Hz 1:1 high definition video signal. It will beappreciated that there will be a 4% speed error in the signals outputfrom the converters 128 and 130, due to the input frame frequency being24 Hz, rather than 25 Hz.

Further secondary output patIls are provided from the VTR 120 via anHDVS to NTSC down converter 132, which produces an NTSC 59.94 Hz 2:1signal. Furthermore, there is a direct output of the 60 Hz 3232 formatsignal on line 134. Also, a standards converter 136 is included forconverting the 60 Hz 3232 format signal from the VTR 120 to a 60 field/s2:1 interlace HDVS signal. It will therefore be appreciated that thestandards converter 136 is as described above with reference to FIG. 56.The output of the standards converter 136 may also be fed to an HDVS toNTSC down converter 138 which produces an NTSC 59.94 Hz 2:1 signal whichwill be of more acceptable quality than the signal provided by theconverter 132, due to proper removal of the phantom fields by thestandards converter 136.

The above arrangement tlas the following features and advantages.Firstly, there is a single stage of motion compensated interpolation, instandards converter 114, between all acquisition media (film 100 andcamera 110) and the primary distribution path on 24 Hz film. Secondly,the post production system 118 allows integration of 60 field/s 2:1 HDVSoriginated material from the camera 110 or other such source withmaterial originated from 24 Hz film. Thirdly, there is provided asecondary distribution route as a 60 Hz 3232 format HDVS signal on line134 which may well be acceptable for many applications. This obviatesthe requirement to post produce video and film release versions ofmaterial separately. Fourthly, there is an additional or alternativesecondary distribution route by way of the 60 field/s HDVS signal fromthe standards converter 136. Lastly, means are provided for convertingthe video signal to conventional definition NTSC (for use in U.S.A. andJapan) and for converting to both high definition and 625 lines signalsat 25 field/s with 2:1 interlace.

There now follows a description of a particular system employing theapparatus described above which is used primarily for transfer from 24Hz film to 60 field/s 2:1 interlaced HDVS permitting HDVS postproduction. Again, due to the complexity of motion compensatedinterpolation processing, the equipment therefor tends to be large andexpensive. In addition, there is always a risk that processing artifactsmay be introduced into the video signal. For these reasons it isimportant that only a single stage of motion compensated processingshould be used between the source(s) and primary distribution path, ifat all possible.

Referring to FIG. 67, the first source is 24 Hz film material 140, andthe primary distribution is by way of a 60 field/s 2:1 HDVS signal online 142. The source film 140 is read by a high definition film scanner144, which provides a 24 Hz 1:1 signal to a VTR 146. The signalreproduced hy the VTR 146 is fed to a standards converter 148 whichconverts the 24 Hz 1:1 format signal to 60 field/s 2:1 interlaced formatsignal, which is recorded by a VTR 150. It will therefore be appreciatedthat the sLandards converter 148 is as described above with reference toFIG. 55.

Second and third sources are in the form of a camera 152 and a VTR 154,each of which produce 60 field/s 2:1 interlaced video signals. Thesignals from the VTR 150, camera 152 and VTR 154 can be integrated inthe HDVS post production system 156 to produce a 60 field/s 2:1interlace HDVS signal, which can be recorded by the VTR 158. The primaryoutput path is directly from the VTR 158 on line 148, which carries thereproduced 60 field/s 2:1 interlaced HDVS.

Secondary distribution paths are provided from the VTR 158 via an HDVSto NTSC down converter 160 which outputs a sLandard NTSC signal and viaan HDVS to 1250/50 high definition converter 162 which provides a 50frame/s 1:1 video signal. A 625 lines 50 field per second 2:1 interlacedsignal can either be provided by a 1250/50 to 625 lines converter 164connected to the output of the converter 162, or by an HDVS to 625/50down converter 166 receiving the 60 field/s 2:1 interlaced HDVS signalfrom the VTR 158.

A further secondary distribution path is provided by a standardsconverter 168 which converts the 60 field/s 2:1 interlaced HDVS signalfrom the VTR 158 to 24 Hz 1:1 format which is recorded by the VTR 170.The standards converter 168 is therefore as described above withreference to FIGS. 1 to 48. The reproduced signal from the VTR 170supplies an EBR 172 to produce 24 Hz 1:1 film 174.

A further secondary disl. ribution path is via a standards converter 176which receives the 60 field/s 2:1 interlaced HDVS from the VTR 158 andprovides a 30 Hz 1:1 format signal to a VTR 178. The converter 176 istherefore as described above with reference to FIGS. 57 to 60. The VTR178 reproduces the 30 Hz 1:1 signal to an EBR 180, which produces a 30Hz 1:1 film 182.

The system described with reference to FIG. 67 has the followingfeatures and advantages. Firstly, there is a single stage of motioncompensated interpolation, in the standards converter 148, between allof the acquisition media and the primary distribution path on line 142.Secondly, the system allows post production integration of 60 field/s2:1 interlaced HDVS originated material with 24 Hz film material, andthe camera 152 may be used live into the post production chain. Asecondary distribution path is provided to output 30 Hz film, and inthis case the standards converter 176 may be used without motioncompensated interpolation processing, using only motion adaptiveinterpolation, as described above with reference to FIGS. 57 and 58. Ameans is provided of outputing 24 Hz film, but this does entail a secondstage of motion compensated interpolation. The system also providesmeans for down converting to conventional definition NTSC (for U.S.A.and Japan) and also means for converting to both high definition and 625lines format at 25 Hz frame rate.

There now follows a description of a particular system employing theapparatus described above which is used primarily for transfer from 30Hz film to 30 Hz film permitting HDVS post production. Again, due to thecomplexity of motion compensated interpolation processing, the equipmenttherefor tends to be large and expensive. In addition, there is always arisk that processing artifacts may be introduced into the video signal.For these reasons it is important that the number of stages of motioncompensated process ing between the source(s) and primary distributionpath should be as few as possible.

Referring to FIG. 68, the first source is 30 Hz film material 190, andthe primary output is 30 Hz film 192. The source film 190 is read by ahigh definition film scanner 194, which provides a 30 Hz 1:1 signal to aVTR 196.

Second and third sources are in the form of a camera 198 and a VTR 200,each of which produce 30 Hz 1:1 video signals. Fourth and fifth sourcesare in the form of a camera 202 and a VTR 204, each of which provide a60 field/s 2:1 interlace HDVS signal to a standards converter 206 whichconverts the signals to 30 Hz 1:1 format with motion adaptiveinterpolation, but not motion compensated interpolation. The convertor206 is therefore of the form described above with reference to FIGS. 57and 58.

The 30 Hz 1:1 format signals from the VTR 196, camera 198, VTR 200 andstandards converter 206 can be integrated in the HDVS post productionsystem 208 to produce an output 30 Hz 1:1 video signals, which can berecorded by the VTR 210. The primary output path is from the VTR 210 toan EBR 212 which products the output 30 Hz 1:1 film 192.

Secondary distribution paths are provided from the VTR 210 via astandards converLer 214 to a VTR 216. The converter 214 converts theinput 30 Hz 1:1 video signals to 60 field/s 2:1 interlace format, and istherefore of the form described above with reference to FIG. 54. Thesignal reproduced by the VTR 216 can therefore be directly output on theline 218 as a 60 Hz 2:1 interlace HDVS signal, and can also be convertedby converters 220, 222, 224 respectively to: NTSC format; 1250/50format; and 625 lines 50 field/s 2:1 interlace formaL. Furthermore, the60 field/s 2:1 interlace HDVS reproduced by the VTR 216 can be convertedby a standards converter 226, of the type described above with referenceto FIGS. 1 to 48, to 24 Hz 1:1 format for producing, via a VTR 228 andEBR 230, 24 Hz 1:1 film 232. The 24 Hz 1:1 signal reproduced by the VTR228 may also be line rate converted by a converter 234 to produce apseudo 25 Hz frame rate video signal.

The system described with reference to FIG. 68 has the followingfeatures and advantages. Firstly, there are no stages of motioncompensated interpolation between any of the acquisition media 190, 198,200, 202 and 204 and the primary distribution path on 30 Hz film 192.Secondly, the system allows post production integration of videooriginated material with 30 Hz film material, and the camera 198 may beused live in the post production chain. Secondary distribution paths areprovided in 60 field/s 2:4 interlaced format and 59.94 Hz NTSC formatwith acceptable motion characteristics and without complex motioncompensated interpolation. Notion portrayal in the 60 field/s 2:1interlaced HDVS signal and NTSC signal is enhanced by the motioncompensated progressive to interlace conversion by the standardsconverter 214. A means is provided of outputting 24 Hz film, but thisdoes entail a second stage of motion compensated interpolation. Thesystem also provides means for down converting to conventionaldefinition NTSC (for U.S.A. and Japan) and also means for converting toboth high definition and 625 lines format at 25 Hz frame rate. Thestandards converter 206 permits standard HDVS 2:1 interlaced cameras 202and VTRs 204 to be used for video acquisition, but their outputs areconverted to 30 Hz 1:1 format by a motion adaptive process, andtherefore the vertical resolution of moving images will be more limitedthan in the case of the 30 Hz 1:1 camera 198 and VTR 200. The system ofFIG. 68 requires the post production chain to process the images inprogressive scan format and in this connection reference is directed topatent application GB 9048805.3, the content of which is incorporatedherein by reference.

There now follows a description of a particular system employing theapparatus described above which is used primarily for transfer from 30Hz or 60 Hz film to 60 field/s 2:1 interlaced HDVS signals permittingHDVS post production. Again, due to the complexity of motion compensatedinterpolation processing, the equipment therefor tends to be large andexpensive. In addition, there is always a risk that processing artifactsmay be introduced into the video signal. For these reasons it isimportant that not more than one stage of motion compensated processingshould be used between the source(s) and primary distribution path, ifat alll possible.

Referring to FIG. 69, the first source is 30 Hz film material 240 or242; the second source is 60 Hz film material 244; the third and fourthsources are a 60 field/s 2:1 interlace HDVS format camera and VTR 248;and the primary distribution is by way of a 60 field/s 2:1 HDVS on line250.

The 30 Hz source film 242 is read by a high definition film scanner 258,which provides a 30 Hz 1:1 signal to a VTR 260. Upon reproduction of thesignal by the VTR 260, it is converted, not necessarily at real-timerate, to 60 field/s 2:1 interlace HDVS format by a standards converter262, of the type described above with reference to FIG. 54, theconverted signal being recorded by a VTR 264. As an alternative, in thecase that real-time conversion from 30 Hz 1:1 to 60 field/s 2:1interlace format becomes possible, the 30 Hz source film 240 may be readby a high definition film scanner 252, which provides a 30 Hz 1:1 signalto a VTR 254. Upon reproduction of the signal by the VTR 254, it isreal-time converted to 60 field/s 2:1 interlace format HDVS by aconverter 256. In the case of the 60 Hz film 244, it is read by a highdefinition film scanner 266, which incorporates a device 268 forpulldown on every field, so that a 60 Hz 2:1 interlace format HDVSsignal is produced, which is recorded by a VTR 27O.

The 60 Hz 2:1 interlace format video signals from the converter, 256,VTR 264, VTR 270, camera 246 and/or VTR 248 can be integrated in theHDVS post production system 272 to produce a 60 field/s 2:1 interlaceHDVS signal, which can be recorded by the VTR 274. The primary outputpath is directly from the VTR 274 on line 250, which carries thereproduced 60 field/s 2:1 interlaced HDVS signal.

Secondary distribution paths are provided from the VTR 274 via an HDVSto NTSC down converter 276 which outputs a standard NTSC signal and viaan HDVS to 1250/50 high definition converter 278 which provides a 50frame/s 1:1 video signal or 50 field/s 2:1 interlace video signal. A 625lines 50 field per second 2:1 interlaced signal can either be providedby a 1250/50 to 625 lines converter 280 connected to the output of theconverter 278, or by an HDVS to 625/50 down converter 282 receiving the60 field/s 2:1 interlaced signal from the VTR 274.

A further secondary distribution path is provided by a standardsconverter 284 which converts the 60 field/s 2:1 interlaced HDVS signalfrom the VTR 274 to 24 Hz 1:1 format which is recorded by the VTR 286.The standards converter 284 is therefore as described above withreference to FIGS. 1 to 48. TIm reproduced signal from the VTR 286supplies an EBR 288 to produce 24 Hz 1:1 film 290.

A further secondary distribution path is via a standards converter 292which receives the 60 field/s 2:1 interlaced HDVS from the VTR 274 andprovides a 30 Hz 1:1 format signal to a VTR 294. The converter 292 istherefore as described above with reference to FIGS. 57 to 60. The VTR294 reproduces the 30 Hz 1:1 signal to an EBR 296, which produces a 30Hz 1:1 film 298.

The system described with reference to FIG. 69 has the followingfeatures and advantages. Firstly, there is a single stage of motioncompensated interpolation, in the standards converter 262 for the 30 Hzfilm 258, between all of the acquisition media and the primarydistribution path on line 250. Secondly, the system allows postproduction integration of 60 field/s 2:1 interlaced HDVS originatedmaterial with 30 Hz and 60 Hz film material, and the camera 246 may beused live in the post production chain. A secondary distribution path isprovided to output 30 Hz film, and in this case the standards converter284 may be used without motion compensated interpolation processing,using only motion adaptive interpolation, as described above withreference to FIGS. 57 and 58. A means is provided of outputting 24 Hzfilm, but this does entail a second stage of motion compensatedinterpolation. The system also provides means for down converting toconventional definition NTSC (for U.S.A. and Japan) and also means forconverting to both high definition and 625 lines format at 25 Hz framerate.

There now follows a description of a particular system employing theapparatus described above which is used primarily for transfer from 30Hz film to 24 Hz film permitting HDVS post production. Again, due to thecomplexity of motion compensated interpolation processing, the equipmenttherefor tends to be large and expensive. In addition, there is always arisk that processing artifacts may be introduced into the video signal.For these reasons it is important that the number of stages of motioncompensated processing between the source(s) and primary distributionpath should be as few as possible.

Referring to FIG. 70, the first source is 30 Hz film material 300, andthe primary output is 24 Hz film 302. The source film 300 is read by ahigh definition film scanner 304, which provides a 30 Hz 1:1 signal to aVTR 306.

Second and third sources are in the form of a camera 308 and a VTR 310,each of which produce 30 Hz 1:1 video signals. Fourth and fifth sourcesare in the form of a camera 312 and a VTR 314, each of which provide a60 field/s 2:1 interlace HDVS signal to a standards converter 316 whichconverts the signals to 30 Hz 1:1 format with motion adaptiveinterpolation, but not necessarily motion compensated interpolation. Theconverter 316 is therefore of the form descrihed above with reference toFIGS. 57 and 58.

The 30 Hz 1:1 formal signals from the VTR 306, camera 308, VTR 310 andstandards converter 316 can he integrated in the HDVS post productionsystem 318 to produce an output 30 Hz 1:1 video signals, which can berecorded by the VTR 320. The primary output path is from the VTR 320 viaa standard converter 322 which converts to 24 Hz 1:1 format and a VTR324 to an EBR 326 which produces the output 24 Hz 1:1 film 302.

A secondary distribut ion path is provided from the VTR 210 to an EBR328 which produces 30 Hz 1:1 formal film 330. Further secondarydistribution paths are via a standards converter 332 to a VTR 334. Theconverter 332 converts the input 30 Hz 1:1 video signal to 60 field/s2:1 interlace HDVS format, and is therefore of the form described abovewith reference to FIG. 54. The signal reproduced by the VTR 334 cantherefore be directly output on the line 336 as a 60 Hz 2:1 interlaceHDVS signal, and can also be converted by converters 338, 340, 342respectively to: NTSC format; 1250/50 format; and 625 lines 50 field/s2:1 interlace lormat.

The system described with reference to FIG. 70 has the followingfeatures and advantages. Firstly, there is only one stage of motioncompensated interpolation (in converter 332) between any of theacquisition media 300, 308, 310, 312 and 314 and the primarydistribution path on 24 Hz 1:1 film 302, and there is no motioncompensated interpolation between the inputs and the output: 30 Hz 1:1film. Secondly, the system allows post production integration of videooriginated material with 30 Hz film material, and the camera 308 may beused live in the post production chain. Secondary distribution paths areprovided in 60 field/s 2:1 interlaced format and 59.94 Hz NTSC formatwith acceptable motion characteristics and without complex motioncompensated interpolation, Motion portrayal in the 60 field/s 2:1interlaced video signal and NTSC signall is enhanced by the motioncompensated progressive to interlace conversion by the standardsconverter 332. The system also provides means for down converting toconventional definition NTSC (for U.S.A. and Japan) and also means forconverting to both high definition and 625 lines format at 25 Hz framerate. The standards converted 316 permits standard HDVS 2:1 interlacedcameras 312 and VTRs 314 to be used for video acquisition, but theiroutputs are converted to 30 Hz 1:1 format by a motion adaptive process,and therefore the vertical resolution of moving images will be morelimited than in the case of the 30 Hz 1:1 camera 198 and VTR 200, Thesystem of FIG. 70 requires the post production chain to process theimages in progressive scan format and in this connection reference isdirected to patent application GB 9018805.3, the content of whichincorporated herein by reference.

In the arrangement described above with reference to FIG. 2, the VTR 11plays at one-eighth speed into the standards converter 12, and thestandards converter provides ten repeats of each output frame. The framerecorder 13 stores one in every ten of the repeated output frames untilit is full, and the stored frames are then output at normal speed to theVTR 14 which records at normal speed. The material is thereforeconverted in segments, entailing starting, stopping and cuing of both ofthe VTRs 11,14. In order to convert one hour of source material using aframe recorder 13 with a capacity of 256 frames, it is necessary tostart, stop and cue each of the recorders 338 times, and it will berealised that this can cause considerable wear of both recorders.Furthermore, the operations of alternately reading from the VTR 11 andthen recording on the VTR 14, with cuing of both recorders results inthe conversion process being slow. Indeed, in the example given above,although the standards converter processes at one-eighth speed, theconversion of one hour of material would take not 8 hours, but almost 11hours. With smaller capacity frame to recorders 13, the wasted the wouldbe increased.

The arrangement of FIG. 71 will now be described, which is designed toincrease the conversion rate to the maximum possible. Instead of oneframe recorder 13 of 256 frame capacity, two frame recorders 13A, 13Bare provided, each of 128 frame capacity, each receiving the output ofthe sLandarcts converter 12, and each controlled by the systemcontroller 15. The outputs of the frame recorders 13A, 13B are fed to a2:1 digital multiplexer 13C, which selects the output from one or theother of the frame recorders 13 under control of the system controller15 and supplies the selecLed signal to the VTR 14. With thismodification, the source VTR 11 is operated non-stop, and every tenthframe in a series of 1280 frames output by the standards converter arerecorded, alternately 128 frames by one frame recorder 13A and 128frames by the other frame recorder 13B. When one frame recorder isrecording, the other frame recorder has time to play back its stored 128frames to the VTR 14. Thus, in the conversion of 1 hour of material, theVTR 11 starts and stops once, and the VTR 14 starts and stops 675 times,with sufficient the between stops and starts for the VTR 14 to be used.Accordingly, the conversion time for 1 hour of material is 8 hours, aslimited by the processing rate of the standards converter.

As an alternative to the arrangement of FIG. 71, a system as shown inFIG. 2 may be used, but in which a frame recorder 13 is used whichpermits simulaneous recording and playback and which is operable with acyclic frame addressing scheme. Thus, frames from the standardsconverter 12 can be stored al the relatively low rate dictated by theconverter 12, and then, when the frame recorder is nearly full, theframes can be played back to the VTR 14 at the relatively high raterequired by the VTR 14 (while the frame recorder is still being fed bythe converter 12) leaving space in the frame recorder to be over writtenby further input frames. The use of this type of frame recorder reducesthe memory requirement, as compared with the arrangement of FIG. 71, andalso obviates the need for the multiplexer 13C.

The arrangement of FIG. 72 will now be described, which is designed toreduce the amount of wear of the VTRs 11, 14. The arrangement. of FIG.72 is similar to that of FIG. 2, except that the path between the VTR 11and the VTR 14 is either via the standards converter 12 when switches11A, 14A under control of the system controller 15 are each in position"1", or via the frame recorder 13 when the switches 11A, 14A are each inposition "2". Operation is in two phases: phase 1 when the switches arein position 1 and then phase 2 when the switches are in position 2.

In the following description it is considered that the tapes on the VTRs11 , 14 have a number of sequential positions or slots for frames, whichare numbered 0 to 86399, plus some spare, in the case of recording 1hour of frames in 24 Hz 1:1 format, and that the frames of the 1 hour ofmaterial to be recorded in 24 Hz 1:1 format are numbered sequentially 0to 86399. I t is also assumed that the frame recorder has a capacity Cof 253 frames.

In phase 1, the VTR 11 and VTR 14 are operated intermittently andsimultaneously. When VTR 11 is playing, the standards converter 12produces a series of frames each repeated R times where R=10. The VTR 11is operated so that RC(R+1)=27830 frames (including the repeats) areproduced by the standards converter 12, and one in every R(=10) of theseframes is recorded by the VTR 14 at normal speed starting at frame slot0, so that frames 0 to 2782 of the material are recorded in frame slots0, 10, 20 . . . 27820 on the tape on the VTR with the intermediate frameslots left blank. The VTR is then cued back to start reading again at aframe slot offset from the previous starting frame slot by C(R+1)=2783frames, so that for this second recording run the starting slot is 2783.A furthher 27830 frames (including repeats) are produced by theconverter and one in ten is recorded, Thus at frame slots 2783, 2793,2803 . . . 30603 of the tape on the VTR. Re-cuing and frame productionand recording continues like this, and the table below gives examples ofthe numbers of frames and the frame slots at which they are recorded,after the VTR's 11, 14 have been started and stopped 1+INT(86399/2783)(=32) times and all 86400 frames have been recorded by the VTR 14.

    ______________________________________                                        Pass First Frame No.                Last Frame No.                            No.  Slot                           Slot                                      ______________________________________                                        0      0            1       2   --   2782                                            0            10      20  --  27820                                     1     2783         2784    2785 --   5565                                           2783         2793    2803 --  30603                                     2     5566         5567    5568 --   8348                                           5566         5576    5586 --  33386                                     3     8349         8350    8357 --  11131                                           8349         8359    8369 --  36169                                     4    11132        11133   11134 --  13914                                          11132        11142   11152 --  38952                                     5    13915        13916   13917 --  16697                                          13915        13925   13935 --  41735                                     6    16698        16699   16700 --  19480                                          16698        16708   16718 --  44518                                     7    19481        19482   19483 --  22263                                          19481        19491   19501 --  47301                                     8    22264        22265   22266 --  25046                                          22264        22274   22284 --  50084                                     9    25047        25048   25049 --  27829                                          25047        25057   25067 --  52867                                     10   27830        27831   27832 --  30612                                          27830        27840   27850 --  55650                                     :    :            :       :     --  :                                              :            :       :     --  :                                         31   86273        86274   86275 --  86399                                          86273        86283   86293 --  87533                                     ______________________________________                                    

If the above table is analyzed, it will be noted that any particularframe, having frame number F, is recorded during pass numberP=INT(F/C(R+1)), and that it is recorded at slot number S=RF-CP(R² -1).It, should be also noted that nearer the beginning and end of therecorded tape, not. every frame slot is used. For example, between slots0 and 2783, nine out of ten slots are left blank, and between slots 2783and 5566 eight out of ten are blank.

In the second phase, the tape recorded on the VTR 14 during phase 1 isloaded onto the VTR 11; a fresh tape is loaded onto the VTR 14; and theswitches 11A, 14A are moved to position "2". The tapes on the VTRs 11,14 are then run continuously at normal speed and rewound repeatedlyuntil the tapes have made R+1 (=11) passes through the VTRs 11, 14.During each pass, selected frames from the VTR 11 are stored in theframe recorder 13 until it has reached its capacity and the storedframes are then output to and recorded by the VTR 14; and this isrepeated until the end of the recorded tape is reached. For each pass,there is a different offset between the slot numbers of the tapes on thetwo VTRs 11, 14.

More specifically, for each pass P (P=0 to R), the VTR 14 is startedafter the VTR 11 so that there is an offset between the slot number S₁of the tape on the VTR 14 and the slot number S₀ the tape on the VTR 11of S₀ -S₁ =C(PR+R-P). Starting at slot number S₀ =PCR on the VTR 11,every Rth (=10 th) frame is stored in the frame recorder 13 until it isfull, i.e. has stored C frames. With both VTRs 11, 14 running, thestored Iraroes are output from the frame recorder 13 at normal speed andrecorded by the VTR 14. This is then repeated so that every next Rthframe from the VTR 11 is stored in the frame recorder 13, and so on. Aspecific example of this is shown in the table below, where F₀ is theoriginal frame numher of a frame input to the frame recorder and F₁ isthe original frame; number at a frame output from the frame recorder.

    __________________________________________________________________________    S.sub.0                                                                           F.sub.0                                                                           F.sub.1                                                                           S.sub.1                                                                           S.sub.0                                                                           F.sub.0 F.sub.1 S.sub.1                                   __________________________________________________________________________    Pass 0          Pass 1                                                          0   0     --  2530                                                                               253            --                                         10   1     --  2540                                                                               254            --                                         20   2     --  2550                                                                               255            --                                        :   :       --  :   :               --                                        2520                                                                               252    --  5050                                                                               505            --                                        2530      0   0 5060         253     253                                      2531      1   1 5061         254     254                                      :       :   :   :   :       :       :                                         2782     252                                                                               252                                                                              5312         505     505                                      2783                                                                              2783     253                                                                              5313                                                                              3036             506                                      2793                                                                              2784     263                                                                              5323                                                                              3037             507                                      :   :       :   :   :               :                                         5303                                                                              3035    2773                                                                              7833                                                                              3288            3026                                      5313    2783                                                                              2783                                                                              7843        3036    3036                                      5314    2784                                                                              2784                                                                              7844        3037    3037                                      :       :   :   :           :       :                                         5565    3035                                                                              3035                                                                              8095        3288    3288                                      :   :       :   :   :               :                                         :   :       :   :   :               :                                         Pass 10         Pass P (General case)                                         25300                                                                             2530    --  PCR CP                                                        25310                                                                             2531    --  +10  +1                                                       :   :       --  :   :                                                         27820                                                                             2782    --  +10  +1                                                       27830   2530                                                                              2530                                                                              +10         CP      CP                                        27831   2531                                                                              2531                                                                               +1          +1      +1                                       :       :   :   :           :       :                                         28082   2782                                                                              2782                                                                               +1          +1      +1                                       28083                                                                             5313    2783                                                                               +1 C(P + R + 1)     +1                                       28093                                                                             5314    2793                                                                              +10  +1             +10                                       :   :       :   :   :               :                                         30603                                                                             5565    5303                                                                              +10  +1             +10                                       30613   5313                                                                              5313                                                                              +10         C(P + R + 1)                                                                          +10                                       30614   5314                                                                              5314                                                                               +1          +1      +1                                       :       :   :   :           :       :                                         30865   5565                                                                              5565                                                                               +1          +1      +1                                       :   :       :    +1 C(P + 2R + 2)    +1                                       :   :       :   +10  +1             +10                                                       :   :               :                                         __________________________________________________________________________

From an analysis of the above tables it will be appreciated that theoriginal frames are recorded stage by stage at the appropriate frameslots on the tape on the VTR 14, so Ihat after the R+1 passes all, ofthe original frames are recorded in the correct order on the tape.

For the above system to operate satisfactorily without frames on theintermediate tape being overwritten, it is necessary that (C modulo R)is non-zero, and that (C modulo R) and R do not have a common factor.Values of C and R may be chosen which do not satisfy these conditions,but it is then necessary to employ a more complicated system todetermine the slavting slots of the intermediate tape in phase 1, andthe offsets S₀ -S₁ in phase 2, and possibly to make more than (R+1 )passes of the tapes in phase 2.

During phase 1 of the above procedure, the number of starts and stops ofeach VTR 11 , 14 is 1+ the integral part of ((highest framenumber)/C(R+1)) which is 1+INT(86399/2783)=32 in the example given, andthus 64 total for both VTRs 11, 14. In phase 2, there are (R+1) startsand stops of each VTR, giving 22 for both recorders. Accordingly, thetotal number of starts and stops is 86 over both phases, which comparesfavourably with the figure of 676 for the arrangement of FIG. 2 withoutthis modification.

In the arrangement described above, it is necessary for the VTR 11 to beahle to provide a slow motion output, for example at one-eighth speed inthe case of conversion from 60 field/s 2:1 interlace format to 24 Hz 1:1format. If a versatile system is to be provided capable of convertingbetween a variety of different formats, then the standards converter 13requires a variety of input speeds, e.g. 1/8, 1/10, 1/20 th. In order toobviate the need for a VTR capable of a variety of playback speeds, thearrangement of FIG. 73 may be used. In this arrangement, a framerecorder 11A is placed in the path between the VTR 11 and the standardsconverter 12 under control of the system controller 15. The VTR 11 isoperated at normal speed, and a series of the output frames (e.g. 256)are stored in the frame recorder 11A. When full, the frame recorderoutputs each stored frame to the standards converter 12 the requirednumber of times, for example ten times to simulate one-tenth speed,meanwhile the VTR 11 is cued ready to supply the next 256 frames to theframe recorder 11A once all of the stored frames have been output to thestandards converter.

With the arrangement of FIG. 73, the standards converter cannot beoperated continuously, because the has to be allowed for frame recorder11A to be refreshed. In order to avoid this problem, a specialcyclically addressable frame store as described above may be used, oralternatively the modification as shown in FIG. 74 may be made. In FIG.74, a pair of frame recorders 11B, 11C are provided in parallel, andeach of which can output via a 2:1 digital multiplexer 11D (controlledby the system controller 15) to the standards converter 12. Thus, wheneither frame recorder 11B, 11C is outputting to the standards converter12, there is the got the next series of frames Lo be stored in the otherframe recorder. Accordingly, continuous output to, and operation of, thestandards converter 12 is permitted.

It will be appreciated that the modifications described above withreference to FIG. 71 and FIGS. 73 or 74 may be combined to permitcontinuous conversion at less than real-time speed without the need forslow-motion replay.

In a motion compensated interpolating system as described above withreference to FIGS. 1 to 48, only a small number of motion vectors can betested on a pixel-by-pixel basis. For optimum operation of the system itis important that the best vectors are pre-selected for testing by themotion vector selector 46, Techniques using global motion vectors onlyhave proved to be good for many types of picture and techniques usingonly locally derived motion vectors have proved good for certainmaterial. Neither is good for all material,

In order to improve on the system described above, a concept will now bedescribed of dividing a picture up into large subdivisions and thencalculating which motion vectors are detected most frequently in each ofthose subdivisions, This technique is an intermediate technique whichcombines good points from both of the above mentioned approaches.

The technique described above includes the process of counting the mostfrequently detected motion vectors within a given field and making thesetootion vectors available for use throughout the picture, Calculatingthe most frequent vectors over the whole picture area and Then applyingthem in sonic overall vector reduction strategy has the advantage ofproviding likely vectors in areas where results obtained fromimmediately surrounding pixels are inconclusive. The technique describedabove with reference to FIGS. 1 to 48 includes the process of `growing`which is a technique of two-dimensional area summing of correlationsurfaces derived from block matching with those derived from adjacentareas of the picture (as described particularly with reference to FIG.21) to enlarge the area over which the match is performed if the natureof the original surface does not permit a good vector to be calculated.Vector reduction as described before considers pictures in progressivelylarger blocks in order to discover satisfactory vectors to be applied topixels within that region. These blocks start with a single `searcharea` of, for example 32×24 pixels which can then be `grown` in avariety of ways up to a maximum of, for example, 160×72 pixels.Thereafter global vectors are derived from the entire picture area of,for example, 1920×1035 pixels.

The advantage of considering a range of different block sizes whendetermining vectors is that the area which just encompasses a movingobject over two field intervals is nearly optimum for discovering thatmotion vector. Thus small blocks favour small objects and large blocksfavour large objects, such as a panning background.

In the strategy described with reference to FIGS. 1 to 48, no arealarger than 160×72 and smaller than 1920×1035 is considered. However, bysubdividing the picture into a number of regular adjoining oroverlapping areas and deriving an intermediate vector or vectors foreach of those areas in turn it is possible to favour the vectorspertaining to larger objects whose features make reliable detection ofvectors over their whole surface by methods described before difficult,but which are still too small relative to the overall picture to besignificant in a list of global vectors.

The subdivision of the picture area may be done in a number of ways. Forexample, the picture area 350 may be divided into a regular array, forexample a 4×3 array, of intermediate areas 352, as shown in FIG. 76, ora non-regular array, for example a 3×3 array, of intermediate areas, asshown in FIG. 78, with, for example, the centre area 354 smaller thanthe rest so that the intermediate vector or vectors for that area aremore localised than for the other areas. The intermediate areas mayadjoin, but be distinct, as in FIGS. 76 and 78, or they may overlap, asshown by the example areas in FIG. 77. Thus, in FIG. 77, the motionvectors available for an output pixel P which lies both in area 356 and358 are: the global vector(s), the intermediate vector(s) for area 356;the intermediate vector(s) for area 358; and the local vector(s) For thepixel P. Alternatively, as shown in FIG. 75, the intermediate vector(s)which are output for an intermediate area 360 may be calculated using anarea 362 which is larger than and encloses the area 360, or indeed anarea which is smaller than the area 360. These methods shown in FIGS. 75and 77 minimise edge effects caused by small parts of larger objectsextending into adjacent intermediate areas.

In the arrangement described with reference to FIGS. 1 to 48, the motionvector for a pixel is selected from the global vector(s) for the wholepicture and the local vector(s) for the pixel under consideration. Thisscheme is expanded to include the possibility of selecting from one ormore types of intermediate vector for intermediate areas including thepixel under consideration, as shown in FIG. 79. The local vectors 359for the picture are used to form a global vector 362 for a 2×2 array ofintermediate areas; second intermediate vectors 364 for a 3×3 array ofintermediate areas; and third intermediate vectors 366 for a 9×6 arrayof intermediate areas. The global, intermediate and original localvectors for each pixel are then supplied in combination to the motionvector selector 46 (FIG. 4).

In the above description, reference has been made to acquiringprogressive scan format signals from photographic film and to usingoutput progressive scan signals to record photographic film. It will beappreciated that other image sources may be used and other end productsmay be generated. For example, the input images may be computergenerated, or produced by an animation camera, or video equipment.

Reference has also been made above to a 3232 pulldown format. It will beappreciated that other pulldown formats, such as 3223, 2323, or 2332 mayalternatively be used.

Having described preferred embodiments of the inventions with referenceto the accompanying drawings, it is to be understood that the inventionsare not limited to the precise embodiments and that various changes andmodification thereof may he effected by one skilled in the art withoutdeparting from the spirit or scope of the inventions as defined in theappended claims.

What we claim is:
 1. A method of converting an input 60 field/s 2:1interlace scan format digital video signal to an output 24 frame/s 60field/s 3232 pulldown format digital video signal, comprising the stepsof:forming a first series of 60 frame/s progressive scan format framesfrom the fields of the input signal; forming a second series of 24frame/s progressive scan format frames from the first series of framessuch that at least every other frame in the second series of frames isproduced by motion compensated interpolation between a respective pairof successive frames of the first series of frames; forming alternatelyodd and even fields from the second series of frames such that one fieldin every five is a repeat; and forming said output 24 frame/s 60 field/s3232 pulldown format digital video signal from the alternately odd andeven fields formed from the second series of frames.
 2. A method asclaimed in claim 1, wherein alternate frames of the second series areproduced by motion compensated interpolation equally between arespective successive pair of frames of the first series, and theremaining frames of the second series are not produced by motioninterpolated compensation.
 3. A method as claimed in claim 1, whereinthe alternately odd and even fields are supplied at a rate of 5/2 timesthe rate of production of the frames in the second series of frames. 4.A method as claimed in claim 1, further comprising the step of storingthe frames in the second series at the same rate as they are produced,and wherein the alternately odd and even fields are produced from thestored frames such that one field in every five is a repeat.
 5. A methodas claimed in claim 1, wherein each input field is repeated with arepeat rate 4R times, where R is a predetermined integer, a frame isproduced in the first series for every 4R repeated input fields, a frameis produced in the second series for every 10R repeated input fields,and each frame in the second series is repeated with a repeat rate of 5Rtimes.